It's been raining in Seattle for weeks. Hard to believe anyone from here ever "finds their moment in the sun." But it happened. We did. We found a moment in the sun. It came again after weeks of feeling like we'd been living in darkness.
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

A blog where I post weekly lessons on life, love, fear, happiness, silence, meditation, yoga...it's all here!
It's been raining in Seattle for weeks. Hard to believe anyone from here ever "finds their moment in the sun." But it happened. We did. We found a moment in the sun. It came again after weeks of feeling like we'd been living in darkness.

18 comments:
The moon rise was stunning this week! I find the rainy season to be cosy and the perfect time to be curled up with tea and a stack of books.
HI Divatobe-Yes, it was quite an amazing moon rise! It was HUGE! Yes, I agree...Seattle is perfect for reading, writing, creating art, making music, etc. etc. This is the advantage of the weather here..we have time inside and we get creative ^_^!
What a beautiful experience to fully enjoy and be present to the sunshine and to your own happiness. That's a gift.
Hi spirit that moves me,
Yes, there are so many gifts around us. I see them everywhere and I feel so much gratitude. Just to see the moon or clouds or ocean or sun or even rain...all so precious and beautiful...peace to you, Katherine
I'm happy to hear you got some sun after all that rain.
Thanks Mike! I'm happy too...I bet you all have a lot of SUN where you are from..enjoy it!
Every moment is so precious. This is so true and sadly, so easy to forget. Thank you for this beautiful reminder.
Hi Christie,
You are welcome. I need the reminder too. Thanks for reading!
What a great blog! So fascinating! I'm happy to be a new follower!
~Jules @ www.adbits.blogspot.com
Hi Jules-Thanks for the follow and glad you like my blog..there's lots to sift through here as I just got finished writing 365 lessons for 2010. I'll check out your blog now!
Hi Katherine --
I live near San Francisco, and my house is -- as I like to say -- "on the border of the fog belt." The fog comes over the hill from the ocean, and we get a lot of it, especially during the summer months, in our neighborhood. So when we do get the sun -- and some warmth -- I really do enjoy it. I do wish we got more sun where I live.
Hi Mike-For some reason I thought San Fran was sunny all the time...I forgot about the fog! Glad you do get some sun...and you enjoy it! ^_^!
Beautiful post and thank you very much for the reminder. I really needed this today.
P.S. Such a great pic of you and Yoon :-)
Thank you Maria, for reading!!
I love that feeling when the sun breaks through after the rains. You can feel the life and energy coming back in you.
I was glad to take our vacation and have a change of scenery and have constant sun as we have been having many storms here in the north east. I had a chance to meet Yogasavy/Savira too on our vacation so it was made even more special with sun and special people and places!
Love the pic here of you and your husband!
Namaste'
Hi Deanne! Did you go to India?? It's so great you met Savira...another blog friend, Becky, also met her in India! She gets around ^_^! I missed the chance to meet her when she was in Seattle....darn! Anyway, so glad you had a chance to go on a journey..journeys can be so eye-opening and when we are in different spaces, sometimes our soul opens even more. It's easy to get stuck in a rut in the same place. Traveling opens us up...something about the open road and possibilities...lots of love to you! Katherine
Yes we did go to India. I love to travel and have made it to 21 countries in the past 6 years. It has become food for my soul and a unique experience my daughter and I share.
In case you didn't read my first blog back in February of last year I intended this to be a travel blog and it took an unexpected turn but one that I needed to take to heal old wounds.
Anyway we will be going to China and Tibet this summer for three weeks as well so I am very excited.
I was so happy to meet Savira and connect with a fellow blogger and friend! Hopefully we will meet on your book tour someday soon too!!
Lots of Love!
That is so wonderful Deanne....you will love Tibet and China...such beautiful, awe-inspiring places!
Post a Comment