Jtdylktuy

I'm trying to get data from a web-page

data=Import["https://stats.nba.com/team/1610612738/traditional/", "Data"]

It returns only a textual information. {Traditional, Advanced, Four Factors, Misc, Scoring}, {{t.city }} {{ t.name }}, {...} etc.

"FullData"element performs in the same way and adding some empty lists.
Can anyone suggest how to handle this type of things in WM? Many thanks.

Since the page is generated asynchronously, you can use the same data source that the page itself does. Using the network inspector in my browser, I discovered that the page loaded its data from this url. It's some API that delivers JSON, so we can use that data.

The first thing we can do is parse the URL, so get an idea of the various parameters we can send to it:

url = "https://stats.nba.com/stats/teamdashboardbygeneralsplits?DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlusMinus=N&Rank=N&Season=2018-19&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&Split=general&TeamID=1610612738&VsConference=&VsDivision=";

URLParse[url]

<|"Scheme" -> "https", "User" -> None, "Domain" -> "stats.nba.com",
"Port" -> None,
"Path" -> {"", "stats", "teamdashboardbygeneralsplits"},
"Query" -> {"DateFrom" -> "", "DateTo" -> "", "GameSegment" -> "",
"LastNGames" -> "0", "LeagueID" -> "00", "Location" -> "",
"MeasureType" -> "Base", "Month" -> "0", "OpponentTeamID" -> "0",
"Outcome" -> "", "PORound" -> "0", "PaceAdjust" -> "N",
"PerMode" -> "PerGame", "Period" -> "0", "PlusMinus" -> "N",
"Rank" -> "N", "Season" -> "2018-19", "SeasonSegment" -> "",
"SeasonType" -> "Regular Season", "ShotClockRange" -> "",
"Split" -> "general", "TeamID" -> "1610612738",
"VsConference" -> "", "VsDivision" -> ""}, "Fragment" -> None|>

Now, you didn't say what you were looking for specifically, but it's clear enough that it would be possible to modify this (for instance, changing TeamID to get a different team, or DateFrom and DateTo to get specific dates. It's worth using your browser's network inspector while changing things in the web page to see more details about these fields.

Anyway, let's just take the results of the initial URL.

Import[url, "JSON"]

Unfortunately, it seems like Mathematica's JSON parser doesn't like the output of that URL, even though it seems like completely valid JSON to me. Perhaps we've stumbled across a bug.

A way to get around this is to use Python's JSON import instead, so let's do that:

pythonImportJson[jsonText_] := 

 ExternalEvaluate["Python", 

  "import json; json.loads("""" <> jsonText <> """")"]

pythonImportJson[Import[url, "Text"]]

The bug that causes our JSON to not import correctly is that Mathematica fails at importing JSON when the numbers are at a high precision. If we look at the output from that API, we see that many of the numbers have very high precision, for instance numbers like -8.300000000000000000000.

A way to get around this is to replace repeated 0s with an empty string and import the resulting string, or to use the python method above.

ImportString[

 StringReplace[Import[url], Repeated["0", {10, [Infinity]}] -> ""], "JSON"]

Now we have our data in a nice association format.

Now, it seems to me like the structure of the JSON is essentially table rows and headings.

If we look at the contents of any data in the resultSets association, we see headers and rowSet, and name. To me this seems like it's basically describing the exact table we see on the webpage.

We can transform this into a dataset by AssociationThreading the headers onto each row in the rowSets.

headers = data[["resultSets", 1, "headers"]]

rows = data[["resultSets", 1, "rowSet"]]

Dataset[AssociationThread[headers, #] & /@ rows]

Comparing that with the first table on the webpage, it seems like we got it right:

Let's pile this into a function and then try it on the "Days Rest" table, which has a few more rows:

createDataset[resultSet_] := 

 Module[{headers = resultSet["headers"], rows = resultSet["rowSet"]},

  Dataset[AssociationThread[headers, #] & /@ rows]]



createDataset[data[["resultSets", 6]]]

Seems pretty good to me.

Now we can create a workflow where we can easily modify the input parameters:

url = URLBuild[URLBuild[

        <|"Scheme" -> "https", "User" -> None, 

         "Domain" -> "stats.nba.com", "Port" -> None, 

         "Path" -> {"", "stats", "teamdashboardbygeneralsplits"}, 

         "Query" -> {"DateFrom" -> "", "DateTo" -> "", 

           "GameSegment" -> "", "LastNGames" -> "0", 

           "LeagueID" -> "00", "Location" -> "", 

           "MeasureType" -> "Base", "Month" -> "0", 

           "OpponentTeamID" -> "0", "Outcome" -> "", "PORound" -> "0",

            "PaceAdjust" -> "N", "PerMode" -> "PerGame", 

           "Period" -> "0", "PlusMinus" -> "N", "Rank" -> "N", 

           "Season" -> "2018-19", "SeasonSegment" -> "", 

           "SeasonType" -> "Regular Season", "ShotClockRange" -> "", 

           "Split" -> "general", "TeamID" -> "1610612738", 

           "VsConference" -> "", "VsDivision" -> ""}, 

         "Fragment" -> None|>];

data = pythonImportJson[Import[url, "Text"]];



Dataset@AssociationThread[data[["resultSets", All, "name"]], 

  createDataset /@ data["resultSets"]]

Which gives us a nice dataset of each table on the page.

Now it's easy to change the "TeamID" parameter to compare teams for instance.