{"id":354,"date":"2006-12-01T01:01:00","date_gmt":"2006-12-01T09:01:00","guid":{"rendered":"http:\/\/netmesh.info\/jernst\/uncategorized\/dan-pritchett-and-randy-shoup-the-ebay-architecture"},"modified":"2006-12-01T01:01:00","modified_gmt":"2006-12-01T09:01:00","slug":"dan-pritchett-and-randy-shoup-the-ebay-architecture","status":"publish","type":"post","link":"https:\/\/upon2020.com\/blog\/2006\/12\/dan-pritchett-and-randy-shoup-the-ebay-architecture\/","title":{"rendered":"Dan Pritchett and Randy Shoup: The eBay Architecture"},"content":{"rendered":"<p>This week, I attended a very interesting presentation by Dan Pritchett and Randy Shoup, both senior technologists at <a href=\"http:\/\/ebay.com\/\" target=\"_blank\">eBay<\/a>, on eBay&#8217;s architecture. Some of it was as I would have expected, other things were, shall we say, counter-intuitive. Here is a random collection of notes, with some special exclamation marks:<\/p>\n<ul>\n<li>212 million registered users, 1 billion photos<\/li>\n<li>1 billion page views a day, 105 million listings, 2 petabytes of data, 3 billion API calls a month<\/li>\n<li>something like a factor of 35 in page views, e-mails sent, bandwidth from June 1999 to Q3\/2006.<\/li>\n<li>99.94% availability, measured as &quot;all parts of site functional to everybody&quot; vs. at least one part of a site not functional to some users somewhere<\/li>\n<li>15,000 application servers, all J2EE. About 100 groups of functionality aka &#8220;apps&#8221;. Notion of a &#8220;pool&#8221;: &#8220;all the machines that deal with selling&#8221;&#8230; Well over 200 databases.<\/li>\n<li>Everything is planned with the question &#8220;what if load increases by 10x&#8221;. Scaling only horizontal, not vertical: many parallel boxes.<\/li>\n<li>leverages MSXML framework for presentation layer (even in Java)<\/li>\n<li>Oracle databases, WebSphere Java (still 1.3.1)<\/li>\n<li>split databases by primary access path, modulo on a key<\/li>\n<li>every database has at least 3 on-line databases. Distributed over 8 data centers<\/li>\n<li>some database copies run 15 min behind, 4 hours behind<\/li>\n<li><em>no stored procedures. some very simple triggers.<\/em><\/li>\n<li>move cpu-intensive work moved out of the database layer to applications applications layer: <em>referential integrity, joins, sorting done in the application layer<\/em>! Reasoning: app servers are cheap, databases are the bottleneck.<\/li>\n<li>no client-side transactions. no distributed transactions<\/li>\n<li>J2EE: use servlets, JDBC, connection pools (with rewrite). Not much else.<\/li>\n<li>no state information in application tier. transient state maintained in cookie or scratch database<\/li>\n<li>app servers do not talk to each other &#8212; strict layering of architecture<\/li>\n<li>Search, in 2002: 9 hours to update the index running on largest Sun box available &#8212; not keeping up<\/li>\n<li>Average item on site changes its search data 5 times before it is sold (e.g. price), so real-time search results are extremely important.<\/li>\n<li>&#8220;Voyager&#8221;: real-time feeder infrastructure built by eBay.. Uses reliable multicast from primary database to search nodes, in-memory search index, horizontal segmentation, N slices, load-balances over M instances, cache queries<\/li>\n<\/ul>\n<p>There were way more questions by the packed audience of architects and other techies than there was time. Absolutely worth everybody&#8217;s time.<\/p>\n<p>Dan put the slides on his <a href=\"http:\/\/www.addsimplicity.com\/\" target=\"_blank\">blog<\/a>: <a href=\"http:\/\/www.addsimplicity.com\/downloads\/eBaySDForum2006-11-29.pdf\" target=\"_blank\">eBaySDForum2006-11-29.pdf<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This week, I attended a very interesting presentation by Dan Pritchett and Randy Shoup, both senior technologists at eBay, on eBay&#8217;s architecture. Some of it was as I would have expected, other things were, shall we say, counter-intuitive. Here is a random collection of notes, with some special exclamation marks: 212 million registered users, 1&hellip;<\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"webmentions_disabled":false,"footnotes":""},"categories":[59],"tags":[],"class_list":["post-354","post","type-post","status-publish","format-standard","hentry","category-comments","kind-"],"kind":false,"_links":{"self":[{"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/posts\/354","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/comments?post=354"}],"version-history":[{"count":0,"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/posts\/354\/revisions"}],"wp:attachment":[{"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/media?parent=354"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/categories?post=354"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/upon2020.com\/blog\/wp-json\/wp\/v2\/tags?post=354"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}