Ubuntu: How to fix “W: Duplicate sources.list entry”?



Question:

I keep getting this warning whenever I try to run sudo apt-get update.

W: Duplicate sources.list entry http://archive.ubuntu.com/ubuntu/ precise-updates/main i386 Packages (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise-updates_main_binary-i386_Packages)  W: You may want to run apt-get update to correct these problems  

Below is the output from /etc/apt/sources.list file:

deb http://archive.ubuntu.com/ubuntu precise main restricted  deb-src http://archive.ubuntu.com/ubuntu precise main restricted    deb http://archive.ubuntu.com/ubuntu precise-updates main restricted  deb-src http://archive.ubuntu.com/ubuntu precise-updates main restricted    deb http://archive.ubuntu.com/ubuntu precise universe  deb-src http://archive.ubuntu.com/ubuntu precise universe  deb http://archive.ubuntu.com/ubuntu precise-updates universe  deb-src http://archive.ubuntu.com/ubuntu precise-updates universe    deb http://archive.ubuntu.com/ubuntu precise multiverse  deb-src http://archive.ubuntu.com/ubuntu precise multiverse  deb http://archive.ubuntu.com/ubuntu precise-updates multiverse  deb-src http://archive.ubuntu.com/ubuntu precise-updates multiverse    deb http://archive.ubuntu.com/ubuntu precise-security main restricted  deb-src http://archive.ubuntu.com/ubuntu precise-security main restricted  deb http://archive.ubuntu.com/ubuntu precise-security universe  deb-src http://archive.ubuntu.com/ubuntu precise-security universe  deb http://archive.ubuntu.com/ubuntu precise-security multiverse  deb-src http://archive.ubuntu.com/ubuntu precise-security multiverse  

How do I fix it?


Solution:1

Your sources.list really has duplicate entries.

First Let's see the correct form of a repository line or source line:

The correct format of repository source line is

<type of repository>  <location>  <dist-name> <components>   

For example, a repo line can be like this one

  deb http://archive.ubuntu.com/ubuntu precise main  

Here, it means, the repository is for binary packages, which are hosted in http://archive.ubuntu.com/ubuntu and this repository is for Ubuntu precise (12.04) and this repository contains the main (software which are officially supported by Canonical) component.

  • Type: The type can be deb and deb-src. deb means a binary repository where deb-src means a source repository

  • Location: http://archive.ubuntu.com/ubuntu location of the repository.

  • Dist-name: The distribution name of Ubuntu release. For Ubuntu 12.04 it is precise, for 11.10 it is oneiric.
    You can visit the Ubuntu Wiki to view an updated list of Ubuntu releases and their code names.

  • Component: It can be main, universe, multiverse and restricted. These words indicates the level of supports for the packages and the licensing status.

See this page for more information.

Please take note that, you can add one or more component in a line, so "main", "universe", "restricted" and "multiverse" can be in a single line. Also note, Though you add more than one component in a single line, APT system considers them as separate line containing only one component.

So, If your sources.list have a line like this

deb http://archive.ubuntu.com/ubuntu precise universe  

Then it can't have another line like the below (which your files have)

deb http://archive.ubuntu.com/ubuntu precise main universe  

which is equivalent of these two lines

deb http://archive.ubuntu.com/ubuntu precise main  deb http://archive.ubuntu.com/ubuntu precise universe  

Because, you are duplicating universe twice, so there will be an error for that duplicate. An error will be given for each duplicate found.

Solution:

After analyzing your sources.list file, I found that, It is a basic one which is supplied by Ubuntu by default. You can build a default file following the procedure:

  1. Open a terminal and first rename the existing sources.list file to sources.list.bak file. (we can safely remove that, but caution is good).

     sudo mv /etc/apt/sources.list /etc/apt/sources.list.bak  
  2. Then open software-properties-gtk to build a new one. Type the below command in a terminal or in dash command prompt with pressing Alt+F2

    software-properties-gtk  

    Note, for Ubuntu version 11.10 or earlier you have to use gksu before the software-properties-gtk command

  3. After the window is open,

    • Select all four categories in Ubuntu Software tab
    • Select precise-security, precise-updates and precise-backports category in Updates tab.
    • Select Canonical partner and Independent category from Other software tab.

That's it. You have now a default sources.list file without error. Compare this file with the previous one, if you want.


Update to deal with sources.list.d dir's files

Sometimes a duplicate entry can be in a file in the /etc/apt/sources.list.d directory, which is also used by APT. So, you should look at that directory and see if there is any duplicate in those files. It is not necessary for duplicates being in the same file.

Example case:

A user had this error message showing while doing sudo apt-get update.

W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main amd64 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages)  W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main i386 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-i386_Packages)  W: You may want to run apt-get update to correct these problems  

But there was no entry with http://dl.google.com in the main /etc/apt/sources.list file. Looking at the /etc/apt/sources.list.d/ directory we found these files:

output of ls /etc/apt/sources.list.d/:

aims-sagemath-precise.list  aims-sagemath-precise.list.save  dropbox.list  dropbox.list.save  fossfreedom-packagefixes-precise.list  fossfreedom-packagefixes-precise.list.save  google-chrome.list  google-chrome.list.save  google.list  google.list.save  sagemath-monolithic-precise.list  sagemath-monolithic-precise.list.save  

The files google-chrome.list and google.list were good candidates for containing duplicates. So, checking the contents of both files was necessary.

Output of cat /etc/apt/sources.list.d/google-chrome.list:

deb http://dl.google.com/linux/chrome/deb/ stable main    

and of cat /etc/apt/sources.list.d/google.list

deb http://dl.google.com/linux/chrome/deb/ stable main  

So, there was a duplicate entries in the sources of Apt. Since google-chrome.list only contain a single apt line and it was also listed in google.list file we can safely remove this file with the command

sudo rm /etc/apt/sources.list.d/google-chrome.list  

Then the problem was solved.


Solution:2

For Ubuntu 12.04

Graphically:

Press Alt + F2 and paste software-properties-gtk (or you can open "Software Center" then go to "Edit" > "Software Sources"). Go to tab "Other Software", choose the duplicate entry and press the "Remove" button.

enter image description here


If you need a command line option, here it is:

cat /etc/apt/sources.list | perl -ne '$H{$_}++ or print' > /tmp/sources.list && sudo mv /tmp/sources.list /etc/apt/sources.list    

The command does this:

cat reads the file and passes the content to perl which removes the duplicate lines. The result is then saved > in a temporary file which is then moved to replace the original /etc/apt/sources.list file.


Solution:3

How bad are duplicate entries in sources.list?

I don’t know how bad it is, but i don’t like sudo apt-get update showing me duplicate entries.

By the way its not that bad, its just showing you that you have duplicate entries.

SourcesList

The sources.list file is a key factor in adding or upgrading applications to your Ubuntu installation. This is also used by your system for system updates. The file is basically the roadmap for your system to know where it may download programs for installation or upgrade.

Its just like Windows update

You can remove duplicate entries in few easy steps with Y PPA Manager

 sudo add-apt-repository ppa:webupd8team/y-ppa-manager -y   sudo apt-get update   sudo apt-get install y-ppa-manager -y  

Open y-ppa-manager form Dash

Enter You Admin Password

enter image description here

Double Click On Advanced

enter image description here

Scan & Remove Duplicate PPA's & Click Ok

enter image description here

It will take some time ( 1 or 2 Mints ) To Scan & Remove Duplicate PPA's


Solution:4

The partner repository is duplicated inside /etc/apt/sources.list and /etc/apt/sources.list.d/precise-partner.list.

Just remove those "precise-partner" files as the partner repository is already present in sources.list.

sudo rm /etc/apt/sources.list.d/precise-partner.*  


Solution:5

Make a back up copy of your sources.list:

sudo cp /etc/apt/sources.list{,.backup}  

Now remove the original and update:

sudo rm /etc/apt/sources.list && sudo apt-get update  


Solution:6

I had the same problem, open software sources. unchecked "Canonical Partners". The Conflict is between "Canonical Partners" and " Canonical partners Added by software Center"


Solution:7

If you couldn't figure it out in your source.list, then do this:

  1. Left-click on settings icon (at the extreme top corner of your screen) and select System Settings.
  2. Click on Software Sources and move to Other Software.
  3. Uncheck one of the 'Canonical Partners(Source Code) -software packaged by Canonical for their partners' files and one of the 'Canonical Partners -software packaged by Canonical for their partners' files also and click Close to leave.
  4. Simultaneously press CTRL+ALT+T keys to open terminal and type sudo apt-get update.


Solution:8

For Ubuntu 9.10 - 12.04

There is an app called Y-PPA-Manager which can do that and much more.

You can install it like that from a Terminal:

sudo add-apt-repository ppa:webupd8team/y-ppa-manager  sudo apt-get upgrade  sudo apt-get install y-ppa-manager  

After that, just start the application an go to the Advanced menu.

enter image description here

Then select "Scan and remove duplicate PPAs" and hit OK.

enter image description here


Solution:9

10.10 and Earlier

Applications > Ubuntu Software Center > Edit > Software Sources > Other Software

OR

System > Administration > Synaptic Packet Manager > Settings > Repositories

Software Sources config window

Select and Remove duplicate entries.


Solution:10

It sounds like you have two identical lines in your /etc/apt/sources.list file.

Go to terminal, and enter sudo editor /etc/apt/sources.list, then look for any identical lines in that file, and if you find them, delete them. After your done, save and exit, and then do sudo apt-get update and that should resolve your problem.

If that doesn't solve your problem then you're going to have to use a trial and error approach. First, create a backup of your sources.list file, then in the actual file go through and comment out or delete each line, saving and apt-get update-ing after each line has been commented out or deleted. Going through this file line by line commenting out different lines will eventually tell you which line is the superfluous one. Don't forget to uncomment any lines you commented if apt-get update still shows the same prompt, otherwise you'll be left with no sources in your sources.list file.


Solution:11

I searched for such a tool, but I didn't find any...
So, I ended up coding one myself with PHP.

Sorry for the dirty unorganized source code.

chkdup - Screenshot:

enter image description here

Author

Mubarak Alrashidi (DeaDSouL)

Requirements:

  1. PHP.
  2. sudo.

Installation:

  1. Open the terminal.
  2. To install PHP if you don't have it, type: sudo apt-get install php5 php5-cli.
  3. Download the chkdup.
  4. Extract the file anywhere you like.
  5. Type: sudo sh install.sh.

Uninstall:

  1. Open the terminal.
  2. Go to the path you extracted the tool in.
  3. Type: sudo sh uninstall.sh.

How to use it:

Just open the terminal and type sudo chkdup.

FAQs:

What does the install.sh do ?

  • The installer will move the chkdup.php to /opt/ then will create a symbolic link in /usr/bin/.

What about the uninstall.sh ?

  • The uninstaller will remove the following files:

    1. install.sh
    2. /opt/chkdup.php
    3. /usr/bin/chkdup
    4. itself uninstall.sh

Is it safe to use chkdup ?

  • Yes it is, since it takes a backup of the current sources.list before attempting to do anything. But I'm not responsible if anything happened. The full responsibility will be yours.

License

GNU General Public License


Solution:12

Did you check ls /etc/apt/sources.list.d/? Maybe you have some "extra" repositories in there?


Solution:13

Install ubuntu-tweak and remove the duplicate entries from there.

sudo add-apt-repository ppa:tualatrix/ppa  sudo apt-get update  sudo apt-get install ubuntu-tweak  

alt text


Solution:14

No, there isn't a single command line to find and delete duplicated entries in the /etc/apt/sources.list file.

This is because the suggested command lines, which merely duplicate the function of the much simpler built-in uniq command, will only work on a SORTED file and will only remove ADJACENT lines. Furthermore, those commands will only remove lines consisting of an identical string of characters.

Duplicate entries reported by apt-get update will consist of duplicate function entries, such as a repository being included in both its i386 and amd64 variants. The easy and effective way to remove these is to note which repositories are reported as duplicates by apt-get update and remove them via the Software Center. Open it and choose Edit -> Software Sources -> Other Software tab. Simply look for the duplicate entries and uncheck them. (This is also an opportunity to remove any source-code repositories if you're not compiling the packages).

However, it should be noted that apt-get update doesn't only find duplicate entries via sources.list, but includes repository files located in /var/lib/apt/lists/. Deleting duplicate entries from there will only temporarily remove the apt-get update error messages if Ubuntu thinks it needs them, as the files will be automatically re-installed. The fact that apt-get update will report some repositories as duplicates, and then suggest that that you run apt-get update itself to repair them, is a clue that you really don't need to worry about that error message.


Solution:15

In my case the duplicates were in the prerequists-sources.list file. I was able to continue the upgrade by running the following commands in the Terminal:

sudo mv /etc/apt/sources.list.d/prerequists-sources.list /tmp  sudo apt-get update  

Then go to the Update Manager and install the updates.


Solution:16

Well, I solved the Duplicate Entry problem myself by renaming /etc/apt/sources.d (as root) to something else and running sudo apt-get update to re-establish the directory and it's contents.


Solution:17

Went through /etc/apt/sources.list manually sudo nano /etc/apt/sources.list and removed the duplicates. After apt-get update, the problem was solved.


Solution:18

In /etc/apt/sources.list you must comment (add # at the start of the line) the duplicate repository entry and save changes. After that repeat sudo apt-get update.


Solution:19

Use the Ubuntu Sources List Generator.

  • Run sudo -H nautilus in terminal
  • Find /etc/apt/sources.list
  • Right click and open the file with a text editor (if you click on it, it will open the software sources GUI)
  • Simply copy and paste into the sources.list and save.
  • Exit terminal and you're done.

Google seems to have a problem, so I'd suggest leaving that out. I have done this on Ubuntu 12.10 (Studio version with Ubuntu desktop added and Unity for another desktop sign in).


Solution:20

Just edit the file /etc/apt/sources.list (for example use sudo -H gedit /etc/apt/sources.list) and remove the duplicate line or put a # at the start of the line.


Solution:21

I try to complete the possibilities offered from other users using terminal, since that's what you asked, mainly:

log in if you're using a text based install or press Alt+F2 and type gnome-terminal if you're using GUI

Then type

sudo nano /etc/apt/sources.list  

(To understand better: in the directory /etc you can find (nearly) all configuration files of your computer and in the apt directory you find the souces.list file that contains all the repositories that APT uses to update or install and upgrade software)

Edit the file removing or commenting with # at the start of the line any duplicated rows.

Press Ctrl+X to exit and press y to save the file (or n to quit without saving).

Then update to see if you've fixed it:

sudo apt-get update  

Have fun!


Solution:22

S="/etc/apt/sources.list" ;  S2="$S ${S}.d/*.list" ;  grep -b "^deb\`cat $S2 |      grep -i "^deb[[:space:]]http" |      sort |      uniq -dc |      sed -e 's;[[:space:]]\\+[[:digit:]]\\+[[:space:]]\\+deb\\(.\\+$\\);\\1;g'\`$" $S2  

* Line breaks inserted for readability.


Solution:23

Execute the following to remove the duplicates

cat /etc/apt/sources.list | perl -ne '$H{$_}++ or print' > /tmp/sources.list && sudo mv /tmp/sources.list /etc/apt/sources.list    


Solution:24

In the case of W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main amd64 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages)

Open: sources.list.d

rm google.list

so only google-chrome.list remains


Solution:25

It is appropriate job for uniq which omits duplicate lines and prints only unique entries, thus we can take output of uniq and replace old file with uniq's output:

 sort /etc/apt/sources.list | uniq > /tmp/sources && sudo mv /tmp/sources  /etc/apt/sources.list  

Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »