Working with the web has always been accessible thanks in large part to the ability to just inspect the code of whatever site you’re on to see how it works. Many modern sites are being created as single page applications and being driven mainly by client side API calls to retrieve data or update the interface dynamically. This is great boon to the web worker or tinkerer since for each site you’re being handed a method to investigate or consume a websites data at your own pace or in your own preferred method. I’ve written about this in the past championing the open and accessible web over trying to hide your source code. Running your own code in the browser gives you a great deal of power over how you choose to consume or control the content you get. It could be something simple such as wanting to automate Cookie Clicker to work overnight. You can do that easily using small scripts to tell the browser you’re clicking the cookie automatically faster than humanly possible. You might be wondering if you’ve discovered the power to alter prices on ecommerce stores and already started sprouting horns but I’m going to cut you off now.
You only have power over what information has been sent to you or that you have authorization to request so it isn’t hacking. If an item at a garage sale is marked at $5 but you flip it over to find it only cost $2.99 new you’ll have an advantage going into negotiations with the seller. The seller could argue that they didn’t intend on you picking up or examining the item before agreeing to purchase it but they still gave you access to find extra information was available just out of view. If the seller had taken the back sticker off and left it inside their house it isn’t okay to start searching through their home for information outside of what the seller has put out in the sale. For a concrete example you can use a Chrome Extensions that modifies Facebook that gives you a different interface, extra information or perhaps less information to suit your taste. But this only affects your experience of Facebook as you request content from Facebook. No one else that isn’t running that same extension see’s any of the modifications you get to enjoy.
Now I’ve had some experiences recently that have inspired me to do this kind of inspection and investigation into sites I’ve been running across because the choices the site owners make feel hostile to the users. Many sites hide content from you until you’ve signed up, logged in or watched some advertisement. Experts-Exchange is a great example of a site loading all the content a user might want to see but then hiding part of it out of the way to coerce them into signing up. Quora for a very long time wouldn’t show you answers to questions until you’d logged in. These sites are or were using methods to try and give selective access to content which was actually sent to your browser but placed just out of site. By looking into the code and content of what has been sent at your request to your browser you can find extra information. The visible content is what the site thinks you need to know but the rest of the invisible content they send is just as important to them.
My wife and I recently started to use an online auction site called Everything but the House or EBTH for short. It feels a lot like eBay did in the early years but with the added security of the seller just being the website. We saw great pieces of furniture selling for extremely low prices but were lucky enough to see that they needed to be shipped before we’d placed a bid. White glove shipping isn’t cheap and is even more expensive when you have large packages. Several hundred dollars for some items such as $500 to ship a chair. Having looked into the costs of shipping I can now appreciate the cost and just keep looking at other items. It was a pain in the butt to see each item, then calculate the shipping cost to figure out if it was even worth bidding or saving. The situation got worse when the site would advertise a sale as taking place in your own local town but all of the items were being shipped from their headquarters out of state. I felt duped having forfeited $9 I’d paid for a couple books to dodge $50 in shipping costs. I know I can make this a better experience.
Having set out to improve the usability of the site I quickly discovered they had wonderful predictable models for their information and page elements. This allowed me to look at the item page and find out how the shipping cost was retrieved and then move that over to the search page. Each item is inspected, a request for shipping to the zipcode is made and then when the cost is returned it is inserted right into the page making fast browsing much more enjoyable.
As you can see from the screenshot I also added the location where the item currently is, the next lowest bid possible and then a combined total you’d end up paying for that item if you won it at that bid. All this I wrapped up into a Chrome Extension Everything but the House Enchancements. It is far from perfect or polished a project but any item listed on the page gets an attempt to populate this extra information for you. Sometimes items can’t be shipped or some other error happens. I haven’t added in code for any of that. This Chrome extension started out as a bunch of JavaScript code I ran in my console when browsing the search pages of EBTH. I only formalized it enough into this browser extension after EBTH had put on local sales where everything was shipped from out of state. This gives you up front the information that EBTH already sends you but hides just out of sight. It will break as the site is updated and may break completely at some point in the future. However it was fun to make and I hope it provides some utility to other EBTH users.
This isn’t the only site I’ve found useful data in the developer tools. One of the more useful bits of information around is what a particular piece was sold for. 1stdibs shows a not so helpful sold sign but doesn’t list the price it was before it was sold.
However if you look closely in the Chrome Developer Tools window at your network tab you can see the price information is right there! This piece was last listed for $2600.
I hope I’ve inspired you in this post to look under the covers of your web browsing experience. More and more of the web is being setup to use these kinds of APIs which will give everyone more opportunities to change the content to their own liking. A big thing to keep in mind while investigating your favorite sites is that they will always be changing. Some data will be here today and gone tomorrow and there will be nothing you can do to change that. What works today might not work tomorrow so you’ll have to be continually learning, investigating and debugging.