How JavaScript Avoids Race Conditions

Posted Wed Dec 26 @ 09:57:24 AM PDT 2012

One thing I've never understood was how JavaScript can avoid race conditions when running code asynchronously (like in an event, AJAX, setTimeout() or setInterval() callback). Yes, JavaScript is single threaded, but that doesn't guarantee a race condition free environment.

Here's a simple example:

We have a function that will change the state of the flag variable after 30 milliseconds. And we busy wait until that happens. Can the assertion fail? That would occur if JavaScript executes the condition in the while loop, and drops down to the assert. But at that moment, the setTimeout() timer expires, and JavaScript switches contexts and starts executing the callback. The callback changes the flag to false. When it finishes, JavaScript resumes where it left off at the assert...and the assert fails.

Can that possibly happen in JavaScript? No. JavaScript will actually patiently wait for the main thread of execution to go into a wait state before executing any asynchronous callback. So in the above case, the setTimeout() callback will never execute, because the thread is always busy.

Parkinson's Law of Triviality

Posted Sun Dec 16 @ 07:26:27 PM PDT 2012

Anyone in a decision making role should be made aware of Parkinson's Law of Triviality (the cool kids call it "bikeshedding"):

The time spent on any item of the agenda will be in inverse proportion to the sum involved. A reactor is used because it is so vastly expensive and complicated that an average person cannot understand it, so one assumes that those that work on it understand it. On the other hand, everyone can visualize a cheap, simple bicycle shed, so planning one can result in endless discussions because everyone involved wants to add a touch and show personal contribution.

Drupal Considered Awful

Posted Sun Dec 02 @ 03:01:09 PM PDT 2012

I had the privilege of using Drupal for the first time. It’s a PHP based content management system that tries to do everything, and fails miserably. As I read through the documentation I came across this little gem, Is Drupal the right tool for the job:

On the other hand, for certain limited uses, Drupal may not be the best choice:

  • If your only requirement is to write a personal blog…
  • …your only requirement is to create a wiki…
  • If your only requirement is to host discussion forums…

Yes, that’s right. They humbly suggest that if you’re building a blog, wiki, or forum, Drupal is the wrong choice. But Drupal is great for everything else! Not quite.

Caching

For whatever reason, Drupal is unbelievably slow right out of the box. To “solve” the problem, the geniuses working on Drupal decided to cache everything. And I mean everything. Yup, if you use Drupal to build your grandma’s baking store website, you need a cache just like the big boys. A quick look at my Drupal site reveals that I can flush the admin menu, cache tables, menu, class registry, page requisites, theme registry, and token registry caches.

What happens if you don’t cache? Simple pages take seconds to load. On my pretty powerful development machine, some pages would take upwards of 5 or 10 seconds to load when the cache was invalidated. That makes developing on Drupal so much more fun. You make a change, flush a cache, wait 5 or 10 seconds for the page to load, and then realize you flushed the wrong cache. Awesome!

hook_spaghetti()

Because Drupal wants to be good at everything (besides running a blog, wiki, or forum), they needed a way for braindead PHP developers to create “modules” on top of it. Inspired by the breathtaking minimalism and celebrated readability of the goto statement, the Drupal developers devised “the hook”. By creating a function with a certain name, your code will magically execute at some point during the page load (unless of course, it gets cached).

Drupal 7 only has about 350 hooks you need to know, so it’s pretty straightforward to memorize the order in which they get called and what they do. What’s more, every module you install can implement each hook too. So during a page load, the same hook might get called multiple times, and you never know in what order! Exciting!

Arrayly-Stringly Typed

Just about everything in Drupal is an array. Need a form? Make an array. Want to build a menu item? Make an array. Want to render a custom page? Array. Array. Array. Array. And by array, it is an array with numeric keys, or an associative array with string keys. Drupal don’t need no stinkin’ constants. Type safely is for those wimpy enterprise Java developers.

Tables

Drupal makes a new table for just about everything. For example, if you create a form with a first name, and a last name, Drupal creates (at least) two tables. One for the first name, and one for the last name. It sure makes your queries interesting.

MySQL Behaves Like a Gas

Posted Sun Nov 25 @ 04:20:35 PM PDT 2012

Earlier this year I bought a (used) server with 16 gigs of RAM. This was a huge upgrade from my little virtual machine with 1 gig of RAM. I thought the 16 fold increase in memory would last me quite a while. But I was wrong. MySQL obeys Boyle's law: It will expand to fill the available space (i.e. RAM).

Less than 200 megabytes of RAM is available, and half my swap space is being used. Two thirds of the RAM is being consumed by MySQL, and Apache comes in at 17%. Fortunately, my web applications are still snappy. But my ibdata1 file continues to grow (it's 14G now), and eventually RAM will definitely be an issue.

A 32 gig pack of cheap RAM for my DL360 G5 comes in at about $1k (more than I paid for the server)...

My Customers

Posted Wed Nov 14 @ 08:00:11 PM PDT 2012

It has almost been almost 4 years since someone paid to use one of my websites. I still remember who my first customer was: a medical doctor named Greg. I've had thousands of customers since then and the vast majority of them are incredibility gracious and patient with me. I am very fortunate to have such great customers.

But of course, there are always a few bad apples. Today, a (former) customer of mine who used my product for a month decided to file a chargeback with their credit card company. The customer never asked for a refund, or even made contact with me; this AMEX card holder just filed the chargeback. On top of that, she left the reason for the chargeback blank. I know it's not a big deal, but it always gets my blood boiling. I emailed the person and of course, she hasn't responded. It's not even about the money. My payment processor will refund the payment, and charge me $15. I'm out $35 bucks. It just feels like a slap in the face when someone is ungrateful for something you've poured hundreds of hours into.

I had another notable bad customer a few months ago. This lady purchased my product, then emailed me a few times asking why certain features were missing. I tried to explain that those features didn't exist in my product, and that she simply misread my list of features, or confused my application with a different one. I told her I would be more than willing to refund her money if she was not satisfied.

I have made it a habit to always offer a refund if I sense the customer is unhappy. They actually sound surprised when I mention it and they rarely take me up on the offer. But I think they feel safer knowing they can get out of the deal if they want to.

But back to the customer...I didn't hear back from her until a few weeks later. She disputed the payment with PayPal (fortunately, she didn't go through the chargeback process or it would have cost me another $15). She claimed I was running some kind of fraud, and purposely misleading customers (or something to that effect). What a nice lady.

I actually think those were the only two bad customers I've had the pleasure of dealing with this year. Fortunately, there are a thousand good customers to every bad one. I've got it pretty good!

Word Search Algorithm Available

Posted Sun Oct 28 @ 02:20:39 PM PDT 2012

I published my word search algorithm on github. I have yet to build a website based on it though. I hope no one beats me to it.

In my earlier post about the algorithm, I mentioned how easy it was to build...except detecting and eliminating badwords. Because my competitor has a badword filter, I needed one too.

What makes a badword filter so hard is that simply detecting a badword is not enough. You have to make sure that the badword is not a substring of another word (like "ass" in "assessment"). Once you find the badword, there are two cases to consider:

  1. The badword is formed only by real words in the word search. For example, notice how "ASS" is formed on the diagonal from the words APE, SAD and SOUP?
    APEKJS
    XSADXK
    ZBSOUP
    In this case, you have to take one of those words, and move it somewhere else, fill in the blanks with random letters, and check again for badwords.
  2. In the easy case, the badword has a letter in it that was randomly chosen. Replace all those random letters with new random letters, and check again for badwords

It takes a lot of code to do that too. The majority of the code in the algorithm is dedicated just to that.

wkhtmltopdf and UTF-8

Posted Sat Oct 20 @ 04:57:26 PM PDT 2012

wkhtmltopdf is a great (maybe the best?) way to convert HTML documents to PDF. It can easily handle most CSS, and it will even execute JavaScript to produce the final HTML output. I use it on two of my websites, and the only downside I have experienced is that is kind of slow (takes a few seconds to make a PDF). Not a deal breaker though.

Recently, I had a user complain that my PDFs were not displaying Chinese characters. The HTML output would display them, but when the PDF was generated, little boxes, question marks, or slashes would appear in their place. Here's an example:

The interesting thing was that my beta server would display the characters in the PDF, but my production server wouldn't. My beta server runs the desktop version of Ubuntu, but the production server runs the server edition. I figured the server edition was missing some package that would display the characters. The hard part was figuring out which package it was!

After Googling around for an hour, I stumbled upon an Ask Ubuntu question that was very loosely related to my problem. Fortunately, one of the packages mentioned in an answer solved the problem for me. The magical package is: ttf-wqy-microhei

On a related note, if you want to use wkhtmltopdf on your Ubuntu server, and you get error messages like:

./wkhtmltopdf: error while loading shared libraries: libfontconfig.so.1: cannot open shared object file: No such file or directory

or

./wkhtmltopdf: error while loading shared libraries: libXrender.so.1: cannot open shared object file: No such file or directory

Then this should solve the problem:

apt-get install libxrender-dev ttf-wqy-microhei

My Experience with Stripe

Posted Sat Oct 13 @ 09:43:58 AM PDT 2012

Stripe is a payment processor that takes aim at PayPal. PayPal is an awful company to work with, and they really needs some competition so they're forced to improve. I use PayPal to process payments on all my sites and while it works, it is barely tolerable. The experience for customers goes like this:

  1. Fill out account information on my site
  2. Go to another page (on my website) with a button to go to PayPal
  3. Go to the PayPal page (where it badgers you to create an account, even though you don't need one)
  4. Find and click the tiny link that says pay by credit card
  5. Enter in your credit information and submit
  6. Confirm
  7. Look for the tiny link back to my website
  8. Wait for the Instant Payment Notification to come through, so their account gets activated

I signed up for a Stripe account, and within an hour or two, had fully implemented their payment system on two of my websites. Here's how the whole process goes now:

  • Fill out account information on my site
  • Enter credit card information and submit (still on my website)
  • Login

That's pretty simple. But here's the disappointing thing: My conversion rate hasn't changed much since I switched from PayPal:

Website PayPal Stripe
JeopardyLabs 0.2320 0.2060
Testmoz 0.0708 0.0794

It means 0.2320 percent of people who entered in their account information on JeopardyLabs, actually end up going through the entire PayPal checkout process (i.e. I get money from them). But for some reason, only 0.2060 complete the checkout process with the crazy simple Stripe implementation! It's hard to believe it has actually gone down. But in the case of Testmoz, the conversion rate is slightly higher. I have had 826 sign ups since I implemented Stripe, so maybe the sample size is too small. I don't remember anything from my statistics class...

Aside from the disappointing conversion rate (which Stripe can't do anything about), there are some things I'm not happy about with Stripe:

  1. No discount on fees. PayPay drops the fees down to 2.5% + $0.30 if you do $3,000+ in monthly sales. You have to do $1,000,000+/year to get any discount from Stripe.
  2. They keep the entire transaction fee for a refund. PayPal just keeps the $0.30, but Stripe keeps the 2.9% too. Fortunately, I only handle a few refunds a year, and the payment ranges from $10-$50. But if you have to give a refund on an expensive product, that is something to consider.
  3. They won't let you dynamically set the memo line (I don't know if that's what it is called, but it's the line of text a customer would see on their credit card statement). If you sell on a lot of different websites, you have to pick one name to put on the customer's credit card statement. They also force you to put a phone number on the memo line too (Google voice to the rescue). In their defense, they allow you to setup multiple accounts, and each account can have its own memo line.
  4. They require SSL (and they should). But SSL is a pain if you don't have a bunch of IPs available to you, or you don't want to pay for a $500 SAN/UCC certificate!
  5. They only display times in UTC! Sheesh, would it kill them to render times in the user's timezone?

Even with that list of petty nitpicks, compared to PayPal, they come out on top.

Building a Word Search

Posted Sat Oct 06 @ 09:43:32 AM PDT 2012

I decided to build a word search generator. Surprisingly, it only took about an hour to code up my initial version in JavaScript. The gist of the algorithm is like this:

  1. Initialize a 2D array, grid, with n rows and m columns
  2. For each word, word, in the list of words you want to put in the word search:
    1. Randomly pick a row and column (r, c) such that (r, c) < (n, m) (i.e. don't pick something outside the bounds of the array)
    2. Randomly pick a direction, d, in the set {North-To-South, South-To-North, West-To-East, East-To-West, NorthWest-To-SouthEast, SouthEast-To-NorthWest, NorthEast-To-SouthWest, SouthWest-To-NorthEast}
    3. Do a test to see if the word can be placed in the direction d, without overflowing the bounds of the array. For example, if your direction was North-To-South, you would want to make sure word.length + row <= n. For NorthWest-To-SouthEast you would need to test word.length + row <= n and word.length + col <= m
    4. Starting at (r1, c1) = (r, c), for each letter, letter in word:
      1. Make sure grid[r1][c1] is empty or grid[r1][c1] == letter (in this latter case, another letter was placed at grid[r1][c1] but because the letters are the same, there is nothing wrong [words are allowed to intersect on a word search])
      2. Update (r1, c1) appropriately based on the direction, d. For example, if the direction is North-To-South, leave c1 alone, but increment r1. For SouthEast-To-NorthWest do (r1, c1) = (r1 - 1, c1 - 1)
    5. If you get through all the previous tests, you can finally place the word on grid. Starting at (r1, c1) = (r, c), for each letter, letter in word:
      1. Set grid[r1][c1] = letter
      2. Update (r1, c1) appropriately based on the direction, d
  3. Now walk through your entire grid, and if grid[r][c] is empty, put a random letter in that cell

Here is a pretty printed example with the greek alphabet:

 01234567890123456789
0PNHVKLEJDNFZMPLOVOSF
1PCBDQRAPXKQLYXKDOMKM
2GZZZGFGUZFVQRNOEMEGR
3GAMMATVBBLHWXKKLIGQI
4TUSNEJHOOLGNWAUTCAXH
5XCEYVGHEZEXATPNARTZP
6ZUMLHYYNTHCRPPPAODJH
7GIMPCBETAASEPASXNJKG
8NWCVLJWNIJOZHZETATOJ
9DFIITRFHHOKEIYNYKLHS
0DGICHISIGMAALPHAKYRI
1IBURMSNHAUPYMBJPIYED
2SWQEPSILONIYKHQJQCTD
3FZGVULYPICXORFGFDPAZ
4ZHVIQQPZUFPMTYFSHSTH
5HSSCUUHZPDZZOALYLGJE
6ZWHBLAMBDASIBNERMXHM
7PDMPUPSILONSVTAUJIIU
8RHVSUZMBSCZPOMPPZFVX
9YANUXCFKAQFXWQIVOWEI

See any problems (besides my row and column numbering scheme, which is cutoff)? How about the word "HELL" starting at (row, col) = (6, 9) going South-To-North, or "CAWK" at (row, col) = (6, 10) going SouthWest-To-NorthEast? Some 5th grader would spend the whole day looking for words like that in his word search.

Now I'm working on a much harder problem: How to detect and eliminate words like that from generated word searches. I already came up with a clever (if I do say so myself), fast way to detect badwords. Unfortunately, I haven't been able to design a good way to eliminate them from the word search. Just randomly changing the letters in the badword to something else doesn't necessarily work, because it might create another badword somewhere else on the grid (or in the exact same spot).

The Sad State of Banking Passwords

Posted Sun Sep 30 @ 12:35:29 PM PDT 2012

I have financial accounts open at Chase, US Bank, Ally, Vanguard, and Sterling Savings Bank. I thought it would be interesting to list the password policy for each. So here it is:

Chase

  • Must contain 7-32 characters
  • Must include at least one number and one letter
  • Cannot include special characters (&, %, *, etc.)
  • Cannot be the same as your User ID
  • Cannot be the same as any of the last five Passwords you've used

A max length of 32 characters seems a little small, especially since sentence style passwords (like "district9wasthebestmovieevermadeinhistory") are all the rage. But no special characters? That's unacceptable. I can't think of any legitimate reason not to allow special characters. That horrible password policy leads me to believe Chase does not salt and hash passwords.

US Bank

For your protection, passwords must be 8 to 24 characters and include both letters and numbers. Spaces are not allowed. You may also include special characters (such as %, $, &).

Again, we see a pretty conservative length limit. But at least they allow special characters. With that short password length policy, I'd guess they don't salt and hash passwords either.

Ally Bank

The password must be between 8-16 characters long and contain at least one uppercase and one numeric character.

An even more restrictive length. Sheesh. You'd think an extra few bytes for a password wouldn't break the bank.

Vanguard

Your password must have 6 to 10 characters, including at least 2 letters and 2 numbers. Don't use spaces.

10 characters. That's awful.

Sterling Savings Bank

Sterling requires a new Password to be 8 to 32 characters in length, contain at least 1 alpha character and 1 numeric character. It is case sensitive and may not begin or end with a space. Note: The following special characters are allowed: dash (-), dot/period (.), tilde (~), exclamation point (!), at sign (@), pound sign (#), ampersand (&), underscore (_), plus (+), dollar ($). Sterling recommends changing Passwords every 90 days.

They match Chase's length maximum. But at least they allow a handful of special characters.

All in all, these banks' password policies are terrible.

Newer Posts Older Posts


It takes considerable knowledge just to realize the extent of your own ignorance. - Thomas Sowell