In this blog post we cover the process, requirements, and the design of an evaluation framework for NLP and Information Extraction. We cover the reasoning behind such a framework, and discuss its implementation with examples from a Named Entity Recognition evaluation point of view.