{"id":12652,"date":"2022-10-01T11:26:21","date_gmt":"2022-10-01T05:56:21","guid":{"rendered":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/"},"modified":"2022-10-01T11:26:21","modified_gmt":"2022-10-01T05:56:21","slug":"solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate","status":"publish","type":"post","link":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/","title":{"rendered":"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate]"},"content":{"rendered":"<p> [ad_1]<br \/>\n<\/p>\n<div id=\"answer-48484252\" class=\"answer js-answer accepted-answer js-accepted-answer\" data-answerid=\"48484252\" data-parentid=\"48480846\" data-score=\"1\" data-position-on-page=\"1\" data-highest-scored=\"1\" data-question-has-accepted-highest-score=\"1\" itemprop=\"acceptedAnswer\" itemscope itemtype=\"https:\/\/schema.org\/Answer\">\n<div class=\"post-layout\">\n<div class=\"votecell post-layout--left\"><\/div>\n<div class=\"answercell post-layout--right\">\n<div class=\"s-prose js-post-body\" itemprop=\"text\">\n<p>You basically have two steps here: First is exploding the arrays (using the explode functions) to get a row for each value in the array, then fixing each element.<\/p>\n<p>You do not have the schema here so the internal structure of each element in the array is not clear, however, I would assume it is something like a struct with two elements. <\/p>\n<p>This means you would do something like this:<\/p>\n<pre><code>import org.apache.spark.sql.functions.explode\ndf1 = df.withColumn(\"array_elem\", explode(df(\"products\"))\ndf2 = df1.select(\"product_PK\", \"array_elem.*\")\n<\/code><\/pre>\n<p>now all you have to do is rename the columns to the names you need.<\/p>\n<\/p><\/div>\n<div class=\"mt24\"><\/div>\n<\/div>\n<p>            <span class=\"d-none\" itemprop=\"commentCount\"><\/span> <\/p><\/div>\n<\/div>\n<p>[ad_2]<\/p>\n<p>solved How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate] <\/p>\n","protected":false},"excerpt":{"rendered":"<p>[ad_1] You basically have two steps here: First is exploding the arrays (using the explode functions) to get a row for each value in the array, then fixing each element. You do not have the schema here so the internal structure of each element in the array is not clear, however, I would assume it &#8230; <a title=\"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate]\" class=\"read-more\" href=\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/\" aria-label=\"More on [Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate]\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[320],"tags":[803,461,802],"class_list":["post-12652","post","type-post","status-publish","format-standard","hentry","category-solved","tag-apache-spark","tag-dataframe","tag-scala"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate] - JassWeb<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate] - JassWeb\" \/>\n<meta property=\"og:description\" content=\"[ad_1] You basically have two steps here: First is exploding the arrays (using the explode functions) to get a row for each value in the array, then fixing each element. You do not have the schema here so the internal structure of each element in the array is not clear, however, I would assume it ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/\" \/>\n<meta property=\"og:site_name\" content=\"JassWeb\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-01T05:56:21+00:00\" \/>\n<meta name=\"author\" content=\"Kirat\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kirat\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/\"},\"author\":{\"name\":\"Kirat\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\"},\"headline\":\"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate]\",\"datePublished\":\"2022-10-01T05:56:21+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/\"},\"wordCount\":113,\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"keywords\":[\"apache-spark\",\"dataframe\",\"scala\"],\"articleSection\":[\"Solved\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/\",\"url\":\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/\",\"name\":\"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate] - JassWeb\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#website\"},\"datePublished\":\"2022-10-01T05:56:21+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/jassweb.com\/solved\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate]\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/jassweb.com\/solved\/#website\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"name\":\"JassWeb\",\"description\":\"Build High-quality Websites\",\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/jassweb.com\/solved\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\",\"name\":\"Jass Web\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"contentUrl\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"width\":693,\"height\":132,\"caption\":\"Jass Web\"},\"image\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\",\"name\":\"Kirat\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"contentUrl\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"caption\":\"Kirat\"},\"sameAs\":[\"http:\/\/jassweb.com\"],\"url\":\"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate] - JassWeb","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/","og_locale":"en_US","og_type":"article","og_title":"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate] - JassWeb","og_description":"[ad_1] You basically have two steps here: First is exploding the arrays (using the explode functions) to get a row for each value in the array, then fixing each element. You do not have the schema here so the internal structure of each element in the array is not clear, however, I would assume it ... Read more","og_url":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/","og_site_name":"JassWeb","article_published_time":"2022-10-01T05:56:21+00:00","author":"Kirat","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kirat","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/#article","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/"},"author":{"name":"Kirat","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31"},"headline":"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate]","datePublished":"2022-10-01T05:56:21+00:00","mainEntityOfPage":{"@id":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/"},"wordCount":113,"publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"keywords":["apache-spark","dataframe","scala"],"articleSection":["Solved"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/","url":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/","name":"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate] - JassWeb","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/#website"},"datePublished":"2022-10-01T05:56:21+00:00","breadcrumb":{"@id":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/jassweb.com\/solved\/solved-how-to-extract-sub-elements-from-the-column-of-dataframe-in-spark-2-duplicate\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/jassweb.com\/solved\/"},{"@type":"ListItem","position":2,"name":"[Solved] How to extract sub-elements from the column of DataFrame in Spark 2? [duplicate]"}]},{"@type":"WebSite","@id":"https:\/\/jassweb.com\/solved\/#website","url":"https:\/\/jassweb.com\/solved\/","name":"JassWeb","description":"Build High-quality Websites","publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/jassweb.com\/solved\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/jassweb.com\/solved\/#organization","name":"Jass Web","url":"https:\/\/jassweb.com\/solved\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/","url":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","contentUrl":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","width":693,"height":132,"caption":"Jass Web"},"image":{"@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31","name":"Kirat","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/","url":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","contentUrl":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","caption":"Kirat"},"sameAs":["http:\/\/jassweb.com"],"url":"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/12652","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/comments?post=12652"}],"version-history":[{"count":0,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/12652\/revisions"}],"wp:attachment":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/media?parent=12652"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/categories?post=12652"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/tags?post=12652"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}