How do I figure out if pursuing patent protection is worth it?

Agreed with previous two posters. One additional comment is that if your patentable invention is nearly impossible to detect (say if someone offered a similar product), then you may be better off with keeping it all a trade secret. Remember, if you do file a patent, you must disclose everything needed by one of average skill in the art to make the invention and that's something that is not right for everyone.

Wether it would be worth to pursue patent protection is - in principle - a business decision where many considerations may come together. You have to keep in mind that patent protection is meant to give you a competitive edge over your competitors. This can only be gained if the invention gives such advantages that customers are willing to pay extra for it (i.e. what is the value of your invention in terms of profit or turnover?) and if it is possible to deter your competitors (are there easy work-arounds, what are the alternatives? Do these minimise your profit or turnover). Next to that come the questions of costs that you need to spend for patent protection (in which countries?) and how high the chances are to obtain a patent (is your invention patent eligible and new and non-Obvious?).
Since the answers to all of these questions are very case-specific, the answers can only be provided after carful advice kowing all the ins and outs of the invention and your business situation. It simply can not be answered on fora like this one.

Interesting question but as Bart van Wezenbeek said, it is a business decision. A business plan should have been done to calculate the cost and the ROI before you go into production. Having an application be denied does not stop you from using Patent Pending status until it has been denied, thereby giving you protection while you are working on it. As you stated, without the protection, you lose the competitive edge and it will also be harder to get funding without some protection.