Resource-Zone: So when should one report dead links (cutoff date) - Resource-Zone

Jump to content

Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

So when should one report dead links (cutoff date) Rate Topic: -----

#1 User is offline   yum-jelly 

  • Member
  • Group: Inactive
  • Posts: 5
  • Joined: 25-November 05

Posted 12 April 2006 - 10:03 AM

Hi

Just one question...

I keep a running count of links that are reported dead. Before I do that, I put them in my DNS cache and perform random lookup (different DNS(s)) for the first 28 days. After I try to connect for 21 more days, 4 times a day. The last week I go back to 8 random DNS checks. After that I remove them from the cache.

Now I do this because I don't want to kill any user reported dead listings that may be because of the users name servers not finding the site.So how should I report these, I have over 40,000, that have been unresponsive for 60 days, and over 90,000, that have been unresponsive for over 90 days.

I have them in this format, but I can format them any way you would like!

These are all still in the April 06, 2006 - dump

www.robustos.com','http://www.robustos.com/','Robusto's Cigar Humidors','Suppliers of wholesale direct storage solutions to consumers and the industry.','Top/Shopping/Tobacco/Cigars/Humidors

www.puroreycigars.com','http://www.puroreycigars.com/','Puro Rey Cigars','Cuban cigars direct from the Caribbean - Habanos, SA authorized dealer.','Top/Shopping/Tobacco/Cigars/Cuban

www.purosimple.com','http://www.purosimple.com/','Puro & Simple','Based in Montreal Canada and featuring a selection of cigars and accessories.','Top/Shopping/Tobacco/Cigars/Cuban

www.echallis.com','http://www.echallis.com/','eChallis','Collection of sarees, dupattas, salwar suits, pretlines, hipsters, skirts, capris and fashion accessories.','Top/Shopping/Clothing/Ethnic and Regional/Asian/Indian

www.fareastbatik.com','http://www.fareastbatik.com/','Far East Batik','Indonesian Batik clothing, including pants, dresses, shirts and robes.','Top/Shopping/Clothing/Ethnic and Regional/Asian/Batik

www.desertimpressions.com','http://www.desertimpressions.com/','Desert Impressions','Custom and preprinted Southwest and Indian motifs, an assortment of our regional wildlife, hot air balloons,water sports, and children's t-shirts.','Top/Shopping/Clothing/Ethnic and Regional/North American

www.weefurbished.com','http://www.weefurbished.com/','A Weefurbished Womb','Clothes in sizes newborn to pre-teen, diapers, gifts, toys and maternity wear.','Top/Shopping/Clothing/Children's/Baby

www.PanHandleBabies.com','http://www.PanHandleBabies.com/','PanHandleBabies','Handmade baby lounge wear for the newborns up to 12 pounds.','Top/Shopping/Clothing/Children's/Baby

preemies2kidsclothing.com','http://preemies2kidsclothing.com/','Sweet Child of Mine','Specializing in clothing for preemies 4 pounds and up, along with infant and toddler.','Top/Shopping/Clothing/Children's/Baby

perikitafashion.com','http://perikitafashion.com/','Perikita','Offers Brazilian cut bikinis for both retail and wholesale markets.','Top/Shopping/Clothing/Swimwear/Bikinis

www.scrubsprn.com','http://www.scrubsprn.com/','Scrubs prn','Offers scrubs in a variety of colors and prints. Also offers shoes, and health care products.','Top/Shopping/Clothing/Uniforms/Medical

www.trendyscrubs.com','http://www.trendyscrubs.com/','Hampton Scrubs','Offering medical uniforms, scrubs for men and women.','Top/Shopping/Clothing/Uniforms/Medical

www.rkuniforms.com','http://www.rkuniforms.com/','RK Uniforms','Authorized dealer of uniforms and accessories for the New York Police Department.','Top/Shopping/Clothing/Uniforms/Public Safety

www.countrygoers.com','http://www.countrygoers.com/','Countrygoers','British designer collection of original clothes in natural fabrics.','Top/Shopping/Clothing/Women's

www.pinkc.com','http://www.pinkc.com/','Pink C','Clothes and accessories.','Top/Shopping/Clothing/Women's

www.scallywag.uk.com','http://www.scallywag.uk.com/','Scallywag','Features tie dye velvet clothing in sizes from small through to extra large.','Top/Shopping/Clothing/Women's

www.expectingmaternitywear.co.uk','http://www.expectingmaternitywear.co.uk/','Expecting Maternity Wear','European fashions for relaxing, working, socializing, and partying. Also carries maternity lingerie.','Top/Shopping/Clothing/Women's/Motherhood/Maternity

www.funkymamamaternity.com','http://www.funkymamamaternity.com/','Funky Mama','Specializing in youthful maternity clothes in unusual designs.','Top/Shopping/Clothing/Women's/Motherhood/Maternity

www.magpieshoard.com','http://www.magpieshoard.com/','Aunty Magpie's Shining Hoard','Designer of coin belts, bra covers, drapes, turban decorations and accessories. Scrunchies and falls made from cowrie shells are also offered.','Top/Shopping/Clothing/Niche/Dance/Middle Eastern/Tribal

www.costumesfromtheattic.com','http://www.costumesfromtheattic.com/','Costumes From The Attic','Specializes in tribal belly dance tassel belts and jewelry.','Top/Shopping/Clothing/Niche/Dance/Middle Eastern/Tribal

www.lvclubwear.com','http://www.lvclubwear.com/','Las Vegas Clubwear','Clubwear for men and women. Dresses, pants, shirts and pimpwear.','Top/Shopping/Clothing/Niche/Genre and Subculture/Clubwear

www.urban-imports.co.uk','http://www.urban-imports.co.uk/','Urban Imports','UK based company stocking imported urban clothing brands including G-Unit, Ecko Unlimited, Rocawear, Phat Farm and Enyce.','Top/Shopping/Clothing/Niche/Genre and Subculture/Hip Hop

blisshandbags.com','http://blisshandbags.com/','Bliss','Handbags and accessories made from natural fibers, vintage, and novelty fabrics.','Top/Shopping/Clothing/Accessories/Handbags and Purses/Fabric and Tapestry/Custom and Handmade

www.kattskreations.com','http://www.kattskreations.com/','Katts Kreations','Handmade handbags, purses, totes and accessories.','Top/Shopping/Clothing/Accessories/Handbags and Purses/Fabric and Tapestry/Custom and Handmade

www.exquisite-styles.com','http://www.exquisite-styles.com/','Exquisite Styles','Retailer of replica handbags and watches inspired by designers.','Top/Shopping/Clothing/Accessories/Handbags and Purses/Designer Inspired

www.getknottedties.com','http://www.getknottedties.com/','Getknottedties.com','A range of formal shirts and ties based in the UK.','Top/Shopping/Clothing/Formal Wear/Men's

www.fashiondish.com','http://www.fashiondish.com/','FashionDish','"Shopping" area of this web-site gives you the chance to buy from up-and-coming designers.','Top/Shopping/Clothing/Formal Wear/Women's/Accessories

www.kidsmoda.com','http://www.kidsmoda.com/','Kidsmoda','Used and new items for infants and toddlers.','Top/Shopping/Clothing/Used and Secondhand/Children

www.angelswardrobe.com','http://www.angelswardrobe.com/','Angels Wardrobe','Offering baby, toddler and older children's clothing.','Top/Shopping/Clothing/Used and Secondhand/Children

www.cateschildrenscostumes.com','http://www.cateschildrenscostumes.com/','Cate's Children's Costumes','Handmade to individual size and specifications. Australia based site offers photos, description, customer comments, and ordering information.','Top/Shopping/Clothing/Costumes/Children's

www.sweatersdirect.com','http://www.sweatersdirect.com/','Irish Sweaters Direct','Luxury Irish knitwear: an exclusive collection of handknit fashion, aran sweaters and cardigans direct from Ireland's finest manufacturers men's, women's and children's.','Top/Shopping/Clothing/Natural Fiber

www.kiwi-collections.com','http://www.kiwi-collections.com/','Kiwi-Collections','Supplier of woollen apparel and shearling footwear.','Top/Shopping/Clothing/Natural Fiber/Wool

www.softsidegirl.com','http://www.softsidegirl.com/','SoftSideGirl.com','Private-label underwear and camisoles for young women.','Top/Shopping/Clothing/Undergarments/Basics

www.fancy-feet.com','http://www.fancy-feet.com/','Fancy Feet','Socks, tights, legwear and accessories for women and girls.','Top/Shopping/Clothing/Undergarments/Socks and Hosiery

www.sockboys.com','http://www.sockboys.com/','Sock Boys','Socks for all occasions.','Top/Shopping/Clothing/Undergarments/Socks and Hosiery

www.poutt.com','http://www.poutt.com/','Poutt.com','Offers glamourous and sexy sleepwear and bras, panties, and robes.','Top/Shopping/Clothing/Undergarments/Lingerie/Exotic

www.uptowngirllingerie.com','http://www.uptowngirllingerie.com/','Uptown Girl Lingerie','Specializes in complete selection of lingerie such as fantasy, bridal, costumes, leather and teddies.','Top/Shopping/Clothing/Undergarments/Lingerie/Exotic

www.jessiejean.com','http://www.jessiejean.com/','Jessie Jean','Features a collection of lingerie, intimate apparel and women's accessories.','Top/Shopping/Clothing/Undergarments/Lingerie/Exotic

wildmountainvision.com','http://wildmountainvision.com/','Wild Mountain Vision','Sweaters, active sports wear, and clothing accessories such as Dale of Norway, Icelandic Design, Tundra Canada, Sierra Designs, and Patagonia.','Top/Shopping/Clothing/Casual

www.glitterheadz.com','http://www.glitterheadz.com/','Glitterheadz','Glitter and rhinestone fashions for girls.','Top/Shopping/Clothing/Casual/T-Shirts





yj!

#2 User is offline   jeanmanco 

  • Member
  • PipPipPip
  • Group: Inactive
  • Posts: 963
  • Joined: 13-August 03
  • Editor Namejeanmanco

Posted 12 April 2006 - 10:10 AM

My goodness! I think this needs a QC tool expert. I will draw attention to your list internally.
Sorry but I do not respond to emails or private messages requesting site review or site status.

#3 User is offline   jeanmanco 

  • Member
  • PipPipPip
  • Group: Inactive
  • Posts: 963
  • Joined: 13-August 03
  • Editor Namejeanmanco

Posted 12 April 2006 - 06:27 PM

Our QC experts suggest waiting for the next run by Robozilla, our link checker. I notice that you reported a lot of dead links last November. Those should have been cleared out by Robozilla in December. You can see the effect of Robozilla in the ODP report for 2005 at http://research.dmoz...report_2005.htm
Sorry but I do not respond to emails or private messages requesting site review or site status.

#4 User is offline   plantrob 

  • Moderator
    DMOZ Admin
  • Group: DMOZ Admin
  • Posts: 68
  • Joined: 29-March 04
  • Editor Nameplantrob

Posted 13 April 2006 - 05:21 AM

Thanks for the report, Yum-Jelly! I've sent you a private message at this board by way of followup.

#5 User is offline   tschild 

  • Moderator
  • Group: Meta
  • Posts: 61
  • Joined: 03-May 02

Posted 13 April 2006 - 08:18 AM

Thanks for the assistance, yum-jelly. I cannot parse "over 40,000, that have been unresponsive for 60 days, and over 90,000, that have been unresponsive for over 90 days." (the first number of sites would have to be be larger, wouldn't it?).

What exactly are "dead links" or "unresponsive sites" in your numbers? Sites that have no DNS entry or are 403s, 404s etc. included?

#6 User is offline   yum-jelly 

  • Member
  • Group: Inactive
  • Posts: 5
  • Joined: 25-November 05

Posted 14 April 2006 - 08:11 AM

Quote

Thanks for the assistance, yum-jelly. I cannot parse "over 40,000, that have been unresponsive for 60 days, and over 90,000, that have been unresponsive for over 90 days." (the first number of sites would have to be be larger, wouldn't it?).

What exactly are "dead links" or "unresponsive sites" in your numbers? Sites that have no DNS entry or are 403s, 404s etc. included?


I have (8) servers in different parts of the world, each have 12 different DNS lookup servers that are updated more than most of the other well known DNS servers. So in this case, a dead link means the URL/Domain does not reply to a request from 96 different DNS servers handling the (host to ip) conversion. So in essence, when user tries to visit the URL/link they get no reply! I handle other HTTP/1.0,1.1 (401,404,..) differently, so they are never in the dead link list!

As for the (60) days list, not having more than the (90) day list...

Each list is maintained separately. Items/domain/URL(s), in the (60) day list are only placed in the (90) day list after they have reached that period of inactivity. So as of today, reported, there are some 150,000 different links that are dead! Granted it might be only 140,000 because some links that are in the April 06, 2006 dump are not listed in the DMOZ. I was going to check each one, but I don't want to overload your servers, doing a scan on each link, to see if it exists in the DMOZ live directory. I think if I did that my IP range would be denied access real quick!


c, ya...

yj!

#7 User is offline   yum-jelly 

  • Member
  • Group: Inactive
  • Posts: 5
  • Joined: 25-November 05

Posted 14 April 2006 - 08:22 AM

RobBroekhuis said:

Thanks for the report, Yum-Jelly! I've sent you a private message at this board by way of followup.


I set you a PM, with links to the data requested...

Take Care

yj!

Share this topic:


Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users


Copyright © 2002 - 2010 Contributing ODP editors - All Rights Reserved.
This site is in no way affiliated with Netscape Communications Corporation. This site is here as a service to directory users and is maintained by a group of editors.