{"id":13727,"date":"2022-10-05T04:41:45","date_gmt":"2022-10-04T23:11:45","guid":{"rendered":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/"},"modified":"2022-10-05T04:41:45","modified_gmt":"2022-10-04T23:11:45","slug":"solved-fastest-way-to-read-file-searching-for-pattern-matches","status":"publish","type":"post","link":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/","title":{"rendered":"[Solved] Fastest way to read file searching for pattern matches"},"content":{"rendered":"<p> [ad_1]<br \/>\n<\/p>\n<div id=\"answer-38228780\" class=\"answer js-answer accepted-answer js-accepted-answer\" data-answerid=\"38228780\" data-parentid=\"38153471\" data-score=\"1\" data-position-on-page=\"1\" data-highest-scored=\"1\" data-question-has-accepted-highest-score=\"1\" itemprop=\"acceptedAnswer\" itemscope itemtype=\"https:\/\/schema.org\/Answer\">\n<div class=\"post-layout\">\n<div class=\"votecell post-layout--left\"><\/div>\n<div class=\"answercell post-layout--right\">\n<div class=\"s-prose js-post-body\" itemprop=\"text\">\n<p>&#8216;grep&#8217; contains decade&#8217;s worth of optimizations, and re-implementing it in any programming language, not just Python, will be slower. *1<\/p>\n<p>Therefore, if speed is important to you, your technique of calling &#8216;grep&#8217; directly is probably the way to go. To do this using &#8216;subprocess&#8217;, without having to write any temporary files, use the &#8216;subprocess.PIPE&#8217; mechanism:<\/p>\n<pre><code>from subprocess import Popen, PIPE\n\nCOMMAND = 'zcat file* | grep oldconfig'\nprocess = Popen(COMMAND, shell=True, stderr=PIPE, stdout=PIPE)\noutput, errors = process.communicate()\nassert process.returncode == 0, process.returncode\nassert errors == '', errors\nprint('{} lines match'.format(len(output.splitlines())))\n<\/code><\/pre>\n<p>This works for me on Python3.5. I&#8217;ve avoided using any of the higher-level interfaces added on top of subprocess recently, so it should work fine on older versions of Python too.<\/p>\n<hr>\n<p>(*1 for example, even with an empty &#8216;for&#8217; loop, as you show in your question, grep is likely to still be faster, because it does not read the input line-by-line. Instead it determine the max number of characters it can seek forwards through the file, ignoring newlines completely, reading one char after each seek, searching for chars that might match any part of the regex. Only if it finds a match does it then look at the characters surrounding that match, to see if the rest of the regex matches and appropriate newlines are present. On top of that it dynamically generates code that is hard-coded to check for matches to the given regex, meaning it executes around 3 x86 instructions per input byte that it examines, and it skips examining most input bytes completely)<\/p>\n<\/p><\/div>\n<div class=\"mt24\"><\/div>\n<\/div>\n<p>            <span class=\"d-none\" itemprop=\"commentCount\"><\/span> <\/p><\/div>\n<\/div>\n<p>[ad_2]<\/p>\n<p>solved Fastest way to read file searching for pattern matches <\/p>\n","protected":false},"excerpt":{"rendered":"<p>[ad_1] &#8216;grep&#8217; contains decade&#8217;s worth of optimizations, and re-implementing it in any programming language, not just Python, will be slower. *1 Therefore, if speed is important to you, your technique of calling &#8216;grep&#8217; directly is probably the way to go. To do this using &#8216;subprocess&#8217;, without having to write any temporary files, use the &#8216;subprocess.PIPE&#8217; &#8230; <a title=\"[Solved] Fastest way to read file searching for pattern matches\" class=\"read-more\" href=\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/\" aria-label=\"More on [Solved] Fastest way to read file searching for pattern matches\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[320],"tags":[1579,349],"class_list":["post-13727","post","type-post","status-publish","format-standard","hentry","category-solved","tag-grep","tag-python"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>[Solved] Fastest way to read file searching for pattern matches - JassWeb<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"[Solved] Fastest way to read file searching for pattern matches - JassWeb\" \/>\n<meta property=\"og:description\" content=\"[ad_1] &#8216;grep&#8217; contains decade&#8217;s worth of optimizations, and re-implementing it in any programming language, not just Python, will be slower. *1 Therefore, if speed is important to you, your technique of calling &#8216;grep&#8217; directly is probably the way to go. To do this using &#8216;subprocess&#8217;, without having to write any temporary files, use the &#8216;subprocess.PIPE&#8217; ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/\" \/>\n<meta property=\"og:site_name\" content=\"JassWeb\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-04T23:11:45+00:00\" \/>\n<meta name=\"author\" content=\"Kirat\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kirat\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/\"},\"author\":{\"name\":\"Kirat\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\"},\"headline\":\"[Solved] Fastest way to read file searching for pattern matches\",\"datePublished\":\"2022-10-04T23:11:45+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/\"},\"wordCount\":243,\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"keywords\":[\"grep\",\"python\"],\"articleSection\":[\"Solved\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/\",\"url\":\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/\",\"name\":\"[Solved] Fastest way to read file searching for pattern matches - JassWeb\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#website\"},\"datePublished\":\"2022-10-04T23:11:45+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/jassweb.com\/solved\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"[Solved] Fastest way to read file searching for pattern matches\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/jassweb.com\/solved\/#website\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"name\":\"JassWeb\",\"description\":\"Build High-quality Websites\",\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/jassweb.com\/solved\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\",\"name\":\"Jass Web\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"contentUrl\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"width\":693,\"height\":132,\"caption\":\"Jass Web\"},\"image\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\",\"name\":\"Kirat\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"contentUrl\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"caption\":\"Kirat\"},\"sameAs\":[\"http:\/\/jassweb.com\"],\"url\":\"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"[Solved] Fastest way to read file searching for pattern matches - JassWeb","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/","og_locale":"en_US","og_type":"article","og_title":"[Solved] Fastest way to read file searching for pattern matches - JassWeb","og_description":"[ad_1] &#8216;grep&#8217; contains decade&#8217;s worth of optimizations, and re-implementing it in any programming language, not just Python, will be slower. *1 Therefore, if speed is important to you, your technique of calling &#8216;grep&#8217; directly is probably the way to go. To do this using &#8216;subprocess&#8217;, without having to write any temporary files, use the &#8216;subprocess.PIPE&#8217; ... Read more","og_url":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/","og_site_name":"JassWeb","article_published_time":"2022-10-04T23:11:45+00:00","author":"Kirat","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kirat","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/#article","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/"},"author":{"name":"Kirat","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31"},"headline":"[Solved] Fastest way to read file searching for pattern matches","datePublished":"2022-10-04T23:11:45+00:00","mainEntityOfPage":{"@id":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/"},"wordCount":243,"publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"keywords":["grep","python"],"articleSection":["Solved"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/","url":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/","name":"[Solved] Fastest way to read file searching for pattern matches - JassWeb","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/#website"},"datePublished":"2022-10-04T23:11:45+00:00","breadcrumb":{"@id":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/jassweb.com\/solved\/solved-fastest-way-to-read-file-searching-for-pattern-matches\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/jassweb.com\/solved\/"},{"@type":"ListItem","position":2,"name":"[Solved] Fastest way to read file searching for pattern matches"}]},{"@type":"WebSite","@id":"https:\/\/jassweb.com\/solved\/#website","url":"https:\/\/jassweb.com\/solved\/","name":"JassWeb","description":"Build High-quality Websites","publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/jassweb.com\/solved\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/jassweb.com\/solved\/#organization","name":"Jass Web","url":"https:\/\/jassweb.com\/solved\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/","url":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","contentUrl":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","width":693,"height":132,"caption":"Jass Web"},"image":{"@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31","name":"Kirat","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/","url":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","contentUrl":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","caption":"Kirat"},"sameAs":["http:\/\/jassweb.com"],"url":"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/13727","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/comments?post=13727"}],"version-history":[{"count":0,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/13727\/revisions"}],"wp:attachment":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/media?parent=13727"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/categories?post=13727"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/tags?post=13727"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}