1<!DOCTYPE html> 2<html lang="en" class="js csstransforms3d"> 3 <head> 4 <meta charset="utf-8"> 5 <meta name="viewport" content="width=device-width, initial-scale=1.0"> 6 <meta name="generator" content="Hugo 0.62.0" /> 7 <meta name="description" content=""> 8 9 10 <link rel="shortcut icon" href="/images/favicon.png" type="image/x-icon" /> 11 12 <title>Querying :: Hue SQL Assistant Documentation</title> 13 14 15 <link href="/css/nucleus.css" rel="stylesheet"> 16 <link href="/css/fontawesome-all.min.css" rel="stylesheet"> 17 <link href="/css/hybrid.css" rel="stylesheet"> 18 <link href="/css/featherlight.min.css" rel="stylesheet"> 19 <link href="/css/perfect-scrollbar.min.css" rel="stylesheet"> 20 <link href="/css/auto-complete.css" rel="stylesheet"> 21 <link href="/css/theme.css" rel="stylesheet"> 22 <link href="/css/hugo-theme.css" rel="stylesheet"> 23 24 <link href="/css/theme-hue.css" rel="stylesheet"> 25 26 27
27<script src="/js/jquery-2.x.min.js"></script>
27 28 29 <style type="text/css"> 30 :root #header + #content > #left > #rlblock_left{ 31 display:none !important; 32 } 33 34 </style> 35 36 </head> 37 <body class="" data-url="/user/querying/"> 38 <nav id="sidebar" class=""> 39 40 41 42 <div id="header-wrapper"> 43 <div id="header"> 44 <a href="http://gethue.com"> 45 <img src="/images/hue_logo.png" alt="alt text" title="Hue Logo"> 46</a> 47 48 </div> 49 50 <div class="searchbox"> 51 <label for="search-by"><i class="fas fa-search"></i></label> 52 <input data-search-input id="search-by" type="search" placeholder="Search..."> 53 <span data-search-clear=""><i class="fas fa-times"></i></span> 54</div> 55
56<script type="text/javascript" src="/js/lunr.min.js"></script>
vendor: 1 bytes, line 56
56
57<script type="text/javascript" src="/js/auto-complete.js"></script>
vendor: 1 bytes, line 57
57
58<script type="text/javascript"> 59 60 var baseurl = "https:\/\/docs.gethue.com\/"; 61 62 var fileSearchBlacklist = ""; 63</script>
vendor: 1 bytes, line 63
63
64<script type="text/javascript" src="/js/search.js"></script>
64 65 66 67 </div> 68 69 <div class="highlightable"> 70 <ul class="topics"> 71 72 73 74 75 76 77 78 79 80 <li data-nav-id="/quickstart/" title="Quick Start" class="dd-item 81 82 83 84 "> 85 <a href="/quickstart/"> 86 Quick Start 87 88 </a> 89 90 91 </li> 92 93 94 95 96 97 98 99 100 101 102 <li data-nav-id="/administrator/" title="Administrator" class="dd-item 103 104 105 106 "> 107 <a href="/administrator/"> 108 <b>1. </b>Administrator 109 110 </a> 111 112 113 <ul> 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 <li data-nav-id="/administrator/installation/" title="Installation" class="dd-item 130 131 132 133 "> 134 <a href="/administrator/installation/"> 135 Installation 136 137 </a> 138 139 140 <ul> 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 <li data-nav-id="/administrator/installation/dependencies/" title="Dependencies" class="dd-item 157 158 159 160 "> 161 <a href="/administrator/installation/dependencies/"> 162 Dependencies 163 164 </a> 165 166 167 </li> 168 169 170 171 172 173 174 175 176 177 178 179 180 <li data-nav-id="/administrator/installation/install/" title="Install" class="dd-item "> 181 <a href="/administrator/installation/install/"> 182 Install 183 184 </a> 185 </li> 186 187 188 189 190 191 192 193 194 195 196 197 198 199 <li data-nav-id="/administrator/installation/cloud/" title="Run" class="dd-item "> 200 <a href="/administrator/installation/cloud/"> 201 Run 202 203 </a> 204 </li> 205 206 207 208 209 210 211 212 </ul> 213 214 </li> 215 216 217 218 219 220 221 222 223 224 225 226 227 <li data-nav-id="/administrator/configuration/" title="Configuration" class="dd-item 228 229 230 231 "> 232 <a href="/administrator/configuration/"> 233 Configuration 234 235 </a> 236 237 238 <ul> 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 <li data-nav-id="/administrator/configuration/server/" title="Server" class="dd-item 255 256 257 258 "> 259 <a href="/administrator/configuration/server/"> 260 Server 261 262 </a> 263 264 265 </li> 266 267 268 269 270 271 272 273 274 275 276 277 278 <li data-nav-id="/administrator/configuration/connectors/" title="Connectors" class="dd-item 279 280 281 282 "> 283 <a href="/administrator/configuration/connectors/"> 284 Connectors 285 286 </a> 287 288 289 </li> 290 291 292 293 294 295 296 297 298 299 300 301 302 <li data-nav-id="/administrator/configuration/apps/" title="Editor" class="dd-item 303 304 305 306 "> 307 <a href="/administrator/configuration/apps/"> 308 Editor 309 310 </a> 311 312 313 </li> 314 315 316 317 318 319 320 </ul> 321 322 </li> 323 324 325 326 327 328 329 330 331 332 333 334 335 <li data-nav-id="/administrator/administration/" title="Administration" class="dd-item 336 337 338 339 "> 340 <a href="/administrator/administration/"> 341 Administration 342 343 </a> 344 345 346 <ul> 347 348 349 350 351 352 353 354 355 356 357 358 359 360 <li data-nav-id="/administrator/administration/reference/" title="Reference" class="dd-item "> 361 <a href="/administrator/administration/reference/"> 362 Reference 363 364 </a> 365 </li> 366 367 368 369 370 371 372 373 374 375 376 377 378 379 <li data-nav-id="/administrator/administration/operations/" title="Operations" class="dd-item "> 380 <a href="/administrator/administration/operations/"> 381 Operations 382 383 </a> 384 </li> 385 386 387 388 389 390 391 392 393 394 395 396 397 398 <li data-nav-id="/administrator/administration/user-management/" title="Users" class="dd-item "> 399 <a href="/administrator/administration/user-management/"> 400 Users 401 402 </a> 403 </li> 404 405 406 407 408 409 410 411 412 413 414 415 416 417 <li data-nav-id="/administrator/administration/database/" title="Database" class="dd-item "> 418 <a href="/administrator/administration/database/"> 419 Database 420 421 </a> 422 </li> 423 424 425 426 427 428 429 430 </ul> 431 432 </li> 433 434 435 436 437 438 439 </ul> 440 441 </li> 442 443 444 445 446 447 448 449 450 451 452 <li data-nav-id="/user/" title="User" class="dd-item 453 parent 454 455 456 "> 457 <a href="/user/"> 458 <b>2. </b>User 459 460 </a> 461 462 463 <ul> 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 <li data-nav-id="/user/concept/" title="Concepts" class="dd-item 480 481 482 483 "> 484 <a href="/user/concept/"> 485 Concepts 486 487 </a> 488 489 490 </li> 491 492 493 494 495 496 497 498 499 500 501 502 503 <li data-nav-id="/user/querying/" title="Querying" class="dd-item 504 parent 505 active 506 507 "> 508 <a href="/user/querying/"> 509 Querying 510 511 </a> 512 513 514 </li> 515 516 517 518 519 520 521 522 523 524 525 526 527 <li data-nav-id="/user/browsing/" title="Browsing" class="dd-item 528 529 530 531 "> 532 <a href="/user/browsing/"> 533 Browsing 534 535 </a> 536 537 538 </li> 539 540 541 542 543 544 545 </ul> 546 547 </li> 548 549 550 551 552 553 554 555 556 557 558 <li data-nav-id="/developer/" title="Developer" class="dd-item 559 560 561 562 "> 563 <a href="/developer/"> 564 <b>3. </b>Developer 565 566 </a> 567 568 569 <ul> 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 <li data-nav-id="/developer/development/" title="Development" class="dd-item 586 587 588 589 "> 590 <a href="/developer/development/"> 591 Development 592 593 </a> 594 595 596 </li> 597 598 599 600 601 602 603 604 605 606 607 608 609 <li data-nav-id="/developer/components/" title="Components" class="dd-item 610 611 612 613 "> 614 <a href="/developer/components/"> 615 Components 616 617 </a> 618 619 620 <ul> 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 <li data-nav-id="/developer/components/scratchpad/" title="SQL Scratchpad" class="dd-item 637 638 639 640 "> 641 <a href="/developer/components/scratchpad/"> 642 SQL Scratchpad 643 644 </a> 645 646 647 </li> 648 649 650 651 652 653 654 655 656 657 658 659 660 <li data-nav-id="/developer/components/parsers/" title="SQL Parsers" class="dd-item 661 662 663 664 "> 665 <a href="/developer/components/parsers/"> 666 SQL Parsers 667 668 </a> 669 670 671 </li> 672 673 674 675 676 677 678 </ul> 679 680 </li> 681 682 683 684 685 686 687 688 689 690 691 692 693 <li data-nav-id="/developer/api/" title="API" class="dd-item 694 695 696 697 "> 698 <a href="/developer/api/"> 699 API 700 701 </a> 702 703 704 <ul> 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 <li data-nav-id="/developer/api/rest/" title="REST" class="dd-item 721 722 723 724 "> 725 <a href="/developer/api/rest/"> 726 REST 727 728 </a> 729 730 731 <ul> 732 733 734 735 736 737 738 739 740 741 742 743 744 745 <li data-nav-id="/developer/api/rest/importer/" title="File Import API" class="dd-item "> 746 <a href="/developer/api/rest/importer/"> 747 File Import API 748 749 </a> 750 </li> 751 752 753 754 755 756 757 758 </ul> 759 760 </li> 761 762 763 764 765 766 767 768 769 770 771 772 773 <li data-nav-id="/developer/api/cli/" title="CLI" class="dd-item 774 775 776 777 "> 778 <a href="/developer/api/cli/"> 779 CLI 780 781 </a> 782 783 784 </li> 785 786 787 788 789 790 791 792 793 794 795 796 797 <li data-nav-id="/developer/api/python/" title="Python" class="dd-item 798 799 800 801 "> 802 <a href="/developer/api/python/"> 803 Python 804 805 </a> 806 807 808 </li> 809 810 811 812 813 814 815 </ul> 816 817 </li> 818 819 820 821 822 823 824 </ul> 825 826 </li> 827 828 829 830 831 832 833 834 835 836 837 <li data-nav-id="/releases/" title="Releases" class="dd-item 838 839 840 841 "> 842 <a href="/releases/"> 843 <b>4. </b>Releases 844 845 </a> 846 847 848 <ul> 849 850 851 852 853 854 855 856 857 858 859 860 861 862 <li data-nav-id="/releases/release-notes-4.11.0/" title="4.11.0" class="dd-item "> 863 <a href="/releases/release-notes-4.11.0/"> 864 4.11.0 865 866 </a> 867 </li> 868 869 870 871 872 873 874 875 876 877 878 879 880 881 <li data-nav-id="/releases/release-notes-4.10.0/" title="4.10.0" class="dd-item "> 882 <a href="/releases/release-notes-4.10.0/"> 883 4.10.0 884 885 </a> 886 </li> 887 888 889 890 891 892 893 894 895 896 897 898 899 900 <li data-nav-id="/releases/release-notes-4.9.0/" title="4.9.0" class="dd-item "> 901 <a href="/releases/release-notes-4.9.0/"> 902 4.9.0 903 904 </a> 905 </li> 906 907 908 909 910 911 912 913 914 915 916 917 918 919 <li data-nav-id="/releases/release-notes-4.8.0/" title="4.8.0" class="dd-item "> 920 <a href="/releases/release-notes-4.8.0/"> 921 4.8.0 922 923 </a> 924 </li> 925 926 927 928 929 930 931 932 933 934 935 936 937 938 <li data-nav-id="/releases/release-notes-4.7.0/" title="4.7.0" class="dd-item "> 939 <a href="/releases/release-notes-4.7.0/"> 940 4.7.0 941 942 </a> 943 </li> 944 945 946 947 948 949 950 951 952 953 954 955 956 957 <li data-nav-id="/releases/release-notes-4.6.0/" title="4.6.0" class="dd-item "> 958 <a href="/releases/release-notes-4.6.0/"> 959 4.6.0 960 961 </a> 962 </li> 963 964 965 966 967 968 969 970 971 972 973 974 975 976 <li data-nav-id="/releases/release-notes-4.5.0/" title="4.5.0" class="dd-item "> 977 <a href="/releases/release-notes-4.5.0/"> 978 4.5.0 979 980 </a> 981 </li> 982 983 984 985 986 987 988 989 990 991 992 993 994 995 <li data-nav-id="/releases/release-notes-4.4.0/" title="4.4.0" class="dd-item "> 996 <a href="/releases/release-notes-4.4.0/"> 997 4.4.0 998 999 </a> 1000 </li> 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 <li data-nav-id="/releases/release-notes-4.3.0/" title="4.3.0" class="dd-item "> 1015 <a href="/releases/release-notes-4.3.0/"> 1016 4.3.0 1017 1018 </a> 1019 </li> 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 <li data-nav-id="/releases/release-notes-4.2.0/" title="4.2.0" class="dd-item "> 1034 <a href="/releases/release-notes-4.2.0/"> 1035 4.2.0 1036 1037 </a> 1038 </li> 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 <li data-nav-id="/releases/release-notes-4.1.0/" title="4.1.0" class="dd-item "> 1053 <a href="/releases/release-notes-4.1.0/"> 1054 4.1.0 1055 1056 </a> 1057 </li> 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 <li data-nav-id="/releases/release-notes-4.0.0/" title="4.0.0" class="dd-item "> 1072 <a href="/releases/release-notes-4.0.0/"> 1073 4.0.0 1074 1075 </a> 1076 </li> 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 <li data-nav-id="/releases/release-notes-3.12.0/" title="3.12.0" class="dd-item "> 1091 <a href="/releases/release-notes-3.12.0/"> 1092 3.12.0 1093 1094 </a> 1095 </li> 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 <li data-nav-id="/releases/release-notes-3.11.0/" title="3.11.0" class="dd-item "> 1110 <a href="/releases/release-notes-3.11.0/"> 1111 3.11.0 1112 1113 </a> 1114 </li> 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 <li data-nav-id="/releases/release-notes-3.10.0/" title="3.10.0" class="dd-item "> 1129 <a href="/releases/release-notes-3.10.0/"> 1130 3.10.0 1131 1132 </a> 1133 </li> 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 <li data-nav-id="/releases/release-notes-3.9.0/" title="3.9.0" class="dd-item "> 1148 <a href="/releases/release-notes-3.9.0/"> 1149 3.9.0 1150 1151 </a> 1152 </li> 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 <li data-nav-id="/releases/release-notes-3.8.0/" title="3.8.0" class="dd-item "> 1167 <a href="/releases/release-notes-3.8.0/"> 1168 3.8.0 1169 1170 </a> 1171 </li> 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 <li data-nav-id="/releases/release-notes-3.7.0/" title="3.7.0" class="dd-item "> 1186 <a href="/releases/release-notes-3.7.0/"> 1187 3.7.0 1188 1189 </a> 1190 </li> 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 <li data-nav-id="/releases/release-notes-3.6.0/" title="3.6.0" class="dd-item "> 1205 <a href="/releases/release-notes-3.6.0/"> 1206 3.6.0 1207 1208 </a> 1209 </li> 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 <li data-nav-id="/releases/release-notes-3.5.0/" title="3.5.0" class="dd-item "> 1224 <a href="/releases/release-notes-3.5.0/"> 1225 3.5.0 1226 1227 </a> 1228 </li> 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 <li data-nav-id="/releases/release-notes-3.0.0/" title="3.3.0" class="dd-item "> 1243 <a href="/releases/release-notes-3.0.0/"> 1244 3.3.0 1245 1246 </a> 1247 </li> 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 <li data-nav-id="/releases/release-notes-2.5.0/" title="2.5.0" class="dd-item "> 1262 <a href="/releases/release-notes-2.5.0/"> 1263 2.5.0 1264 1265 </a> 1266 </li> 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 <li data-nav-id="/releases/release-notes-2.4.0/" title="2.4.0" class="dd-item "> 1281 <a href="/releases/release-notes-2.4.0/"> 1282 2.4.0 1283 1284 </a> 1285 </li> 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 <li data-nav-id="/releases/release-notes-2.3.0/" title="2.3.0" class="dd-item "> 1300 <a href="/releases/release-notes-2.3.0/"> 1301 2.3.0 1302 1303 </a> 1304 </li> 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 <li data-nav-id="/releases/release-notes-2.2.0/" title="2.2.0" class="dd-item "> 1319 <a href="/releases/release
1319-notes-2.2.0/"> 1320 2.2.0 1321 1322 </a> 1323 </li> 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 <li data-nav-id="/releases/release-notes-2.1.0/" title="2.1.0" class="dd-item "> 1338 <a href="/releases/release-notes-2.1.0/"> 1339 2.1.0 1340 1341 </a> 1342 </li> 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 <li data-nav-id="/releases/release-notes-2.0.1/" title="2.0.1" class="dd-item "> 1357 <a href="/releases/release-notes-2.0.1/"> 1358 2.0.1 1359 1360 </a> 1361 </li> 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 <li data-nav-id="/releases/release-notes-2.0.0-beta/" title="2.0.0-beta" class="dd-item "> 1376 <a href="/releases/release-notes-2.0.0-beta/"> 1377 2.0.0-beta 1378 1379 </a> 1380 </li> 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 <li data-nav-id="/releases/release-notes-1.2.0/" title="1.2.0" class="dd-item "> 1395 <a href="/releases/release-notes-1.2.0/"> 1396 1.2.0 1397 1398 </a> 1399 </li> 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 <li data-nav-id="/releases/release-notes-1.1.0/" title="1.1.0" class="dd-item "> 1414 <a href="/releases/release-notes-1.1.0/"> 1415 1.1.0 1416 1417 </a> 1418 </li> 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 <li data-nav-id="/releases/release-notes-1.0.1/" title="1.0.1" class="dd-item "> 1433 <a href="/releases/release-notes-1.0.1/"> 1434 1.0.1 1435 1436 </a> 1437 </li> 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 <li data-nav-id="/releases/release-notes-1.0.0/" title="1.0.0" class="dd-item "> 1452 <a href="/releases/release-notes-1.0.0/"> 1453 1.0.0 1454 1455 </a> 1456 </li> 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 <li data-nav-id="/releases/release-notes-0.9.1/" title="0.9.1" class="dd-item "> 1471 <a href="/releases/release-notes-0.9.1/"> 1472 0.9.1 1473 1474 </a> 1475 </li> 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 <li data-nav-id="/releases/release-notes-0.9.0/" title="0.9.0" class="dd-item "> 1490 <a href="/releases/release-notes-0.9.0/"> 1491 0.9.0 1492 1493 </a> 1494 </li> 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 <li data-nav-id="/releases/release-notes-0.4.2/" title="0.4.2" class="dd-item "> 1509 <a href="/releases/release-notes-0.4.2/"> 1510 0.4.2 1511 1512 </a> 1513 </li> 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 <li data-nav-id="/releases/release-notes-0.4.1/" title="0.4.1" class="dd-item "> 1528 <a href="/releases/release-notes-0.4.1/"> 1529 0.4.1 1530 1531 </a> 1532 </li> 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 <li data-nav-id="/releases/release-notes-0.4.0/" title="0.4.0" class="dd-item "> 1547 <a href="/releases/release-notes-0.4.0/"> 1548 0.4.0 1549 1550 </a> 1551 </li> 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 <li data-nav-id="/releases/release-notes-0.3.0/" title="0.3.0" class="dd-item "> 1566 <a href="/releases/release-notes-0.3.0/"> 1567 0.3.0 1568 1569 </a> 1570 </li> 1571 1572 1573 1574 1575 1576 1577 1578 </ul> 1579 1580 </li> 1581 1582 1583 1584 1585 1586 </ul> 1587 1588 1589 1590 <section id="shortcuts"> 1591 <h3>More</h3> 1592 <ul> 1593 1594 <li> 1595 <a class="padding" href="https://github.com/cloudera/hue"><i class='fab fa-fw fa-github'></i> GitHub</a> 1596 </li> 1597 1598 <li> 1599 <a class="padding" href="http://demo.gethue.com"><i class='fas fa-fw fa-play'></i> Demo</a> 1600 </li> 1601 1602 <li> 1603 <a class="padding" href="http://gethue.com"><i class='fas fa-fw fa-bookmark'></i> gethue.com</a> 1604 </li> 1605 1606 </ul> 1607 </section> 1608 1609 1610 1611 <section id="footer"> 1612 <p></p> 1613 1614 </section> 1615 </div> 1616</nav> 1617 1618 1619 1620 1621 1622 <section id="body"> 1623 <div id="overlay"></div> 1624 <div class="padding highlightable"> 1625 <div> 1626 <div id="top-bar"> 1627 1628 1629 1630 1631 <div id="top-github-link"> 1632 <a class="github-link" title='Edit this page' href="https://github.com/cloudera/hue/blob/master/docs/docs-site/content/user/querying/_index.md" target="blank"> 1633 <i class="fas fa-code-branch"></i>
1634 <span id="top-github-link-text">Edit this page</span> 1635 </a> 1636 </div> 1637 1638 1639 <div id="breadcrumbs" itemscope="" itemtype="http://data-vocabulary.org/Breadcrumb"> 1640 <span id="sidebar-toggle-span"> 1641 <a href="#" id="sidebar-toggle" data-sidebar-toggle=""> 1642 <i class="fas fa-bars"></i> 1643 </a> 1644 </span> 1645 <span class="links"> 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 <a href='/'>Hue Guide</a> > <a href='/user/'>User</a> > Querying 1660 1661 1662 1663 1664 1665 1666 1667 </span> 1668 </div> 1669 </div> 1670 </div> 1671 1672 1673 <div class="body-outer"> 1674 1675 1676 1677<aside class="toc"> 1678 <div class="wrapper"> 1679 <header> 1680 <h4>What's on this Page</h4> 1681 </header> 1682 <nav id="TableOfContents"> 1683 <ul> 1684 <li><a href="#editor">Editor</a> 1685 <ul> 1686 <li><a href="#running-queries">Running Queries</a></li> 1687 <li><a href="#results">Results</a></li> 1688 <li><a href="#autocomplete">Autocomplete</a></li> 1689 <li><a href="#sharing">Sharing</a></li> 1690 <li><a href="#assist">Assist</a></li> 1691 <li><a href="#variables">Variables</a></li> 1692 <li><a href="#charting">Charting</a></li> 1693 <li><a href="#query-troubleshooting">Query troubleshooting</a></li> 1694 <li><a href="#modes">Modes</a></li> 1695 </ul> 1696 </li> 1697 <li><a href="#dashboard">Dashboard</a> 1698 <ul> 1699 <li><a href="#analytics-facets">Analytics facets</a></li> 1700 <li><a href="#autocomplete-1">Autocomplete</a></li> 1701 <li><a href="#marker-map">Marker Map</a></li> 1702 <li><a href="#edit-records">Edit records</a></li> 1703 <li><a href="#links">Links</a></li> 1704 <li><a href="#saved-queries">Saved queries</a></li> 1705 <li><a href="#fixed-or-rolling-window">âFixedâ or ârollingâ window</a></li> 1706 <li><a href="#more-like-this">‘More like this’</a></li> 1707 </ul> 1708 </li> 1709 <li><a href="#notebook">Notebook</a> 1710 <ul> 1711 <li><a href="#spark">Spark</a></li> 1712 <li><a href="#others">Others</a></li> 1713 </ul> 1714 </li> 1715 </ul> 1716</nav> 1717 </div> 1718</aside> 1719 1720 1721 <div id="body-inner"> 1722 1723 1724 <h1>Querying</h1> 1725 1726 1727 1728 1729 1730<p>Hue's goal is to make Databases & Datawarehouses querying easy and productive.</p> 1731<p>Several apps, each one specialized in a certain type of querying are available. Data sources can be explored first via the <a href="/user/browsing/">browsers</a>.</p> 1732<ul> 1733<li>The Editor shines for SQL queries. It comes with an intelligent autocomplete, risk alerts and self service troubleshooting.</li> 1734<li>The Editor is also available in Notebook mode for quickly executing light programming snippets.</li> 1735<li>Dashboards are focusing on visualizing indexed data but can also query SQL databases.</li> 1736</ul> 1737<p>The configuration of the connectors is currently done by the <a href="/administrator/configuration/connectors/">Administrator</a>.</p> 1738<h2 id="editor">Editor</h2> 1739<p><img src="https://cdn.gethue.com/uploads/2019/08/hue_4.5.png" alt="Editor"></p> 1740<h3 id="running-queries">Running Queries</h3> 1741<p>SQL query execution is the primary use case of the Editor. See the list of most common <a href="/administrator/configuration/connectors/">Databases and Datawarehouses</a>.</p> 1742<ol> 1743<li> 1744<p>The currently selected statement has a <strong>left blue</strong> border. To execute a portion of a query, highlight one or more query 1745statements.</p> 1746</li> 1747<li> 1748<p>Click <strong>Execute</strong>. The Query Results window appears</p> 1749<ul> 1750<li>There is a Log caret on the left of the progress bar</li> 1751<li>Expand the <strong>Columns</strong> by clicking on the column label will scroll to the column. Names and types can be filtered</li> 1752<li>Select the <strong>Chart</strong> icon to plot the results</li> 1753<li>To expand a row, click on the row number</li> 1754<li>To lock a row, click on the lock icon in the row number column</li> 1755<li>Search either by clicking on the magnifier icon on the results tab, or pressing <code>Ctrl/Cmd + F</code></li> 1756<li>See more below how to [refine your results](#Refining Results)</li> 1757</ul> 1758</li> 1759<li> 1760<p>If there are <strong>multiple statements</strong> in the query (separated by semi-colons), click Next in the 1761multi-statement query pane to execute the remaining statements.</p> 1762</li> 1763</ol> 1764<p>When you have multiple statements it's enough to put the cursor in the statement you want to execute, the active statement is indicated with a blue gutter marking.</p> 1765<p><strong>Note</strong>: Use <code>CTRL/Cmd + ENTER</code> to execute queries.</p> 1766<p><strong>Note</strong>: On top of the logs panel, there is a link to open the query profile in the <a href="/user/browsing/#sql-queries">Query Browser</a>.</p> 1767<h3 id="results">Results</h3> 1768<h4 id="refining">Refining</h4> 1769<p>Lock some rows: this will help you compare data with other rows. When you hover a row id, you get a new lock icon. If you click on it, the row automatically sticks to the top of the table.</p> 1770<p><img src="https://cdn.gethue.com/uploads/2016/08/lock_rows.gif" alt="Result row locking"></p> 1771<p>The column list follows the result grid, can be filtered by data type and can be resized.</p> 1772<p><img src="https://cdn.gethue.com/uploads/2016/08/column_list.gif" alt="Smart headers"></p> 1773<p>The headers of fields with really long content will follow your scroll position and always be visible.</p> 1774<p><img src="https://cdn.gethue.com/uploads/2016/08/headers.gif" alt="Cell content search"></p> 1775<p>You can now search for certain cell values in the table and the results are highlighted.</p> 1776<p>You can activate the new search either by clicking on the magnifier icon on the results tab, or pressing <code>Ctrl/Cmd + F</code>.</p> 1777<p><img src="https://cdn.gethue.com/uploads/2016/08/search.gif" alt="Virtual cell rendering"></p> 1778<p>The virtual renderer display just the cells you need at that moment.</p> 1779<p>The table you see here has hundreds of columns</p> 1780<p><img src="https://cdn.gethue.com/uploads/2016/08/virtual_renderer.gif" alt="Many cells"></p> 1781<p>If the download to Excel or CSV takes too long, you will have a nice message now</p> 1782<p><img src="https://cdn.gethue.com/uploads/2016/08/downloadwait.gif" alt="Many cells"></p> 1783<h4 id="downloading">Downloading</h4> 1784<p>There are several ways you can export results of a query.</p> 1785<p>The most common:</p> 1786<ul> 1787<li>Download to your computer as a CSV or XLS</li> 1788<li>Copy the currently fetched rows to the clipboard</li> 1789</ul> 1790<p>Two of them offer greater scalability:</p> 1791<ul> 1792<li>Export to an empty folder on your cluster's file system.</li> 1793<li>Export to a table. You can choose an already existing table or a new one.</li> 1794</ul> 1795<p><img src="https://cdn.gethue.com/uploads/2019/04/editor_export_results.png" alt="Downloading and Exporting Results"></p> 1796<h3 id="autocomplete">Autocomplete</h3> 1797<p>To make your SQL editing experience, Hue comes with one of the best SQL autocomplete on the planet. The new autocompleter knows all the ins and outs of the Hive and Impala SQL dialects and will suggest keywords, functions, columns, tables, databases, etc. based on the structure of the statement and the position of the cursor.</p> 1798<p>
1798The result is improved completion throughout. We now have completion for more than just SELECT statements, it will help you with the other DDL and DML statements too, INSERT, CREATE, ALTER, DROP etc.</p> 1799<p><img src="https://cdn.gethue.com/uploads/2017/07/hue_4_assistant_2.gif" alt="Autocomplete and context assist"></p> 1800<p><strong>Smart column suggestions</strong></p> 1801<p>If multiple tables appear in the FROM clause, including derived and joined tables, it will merge the columns from all the tables and add the proper prefixes where needed. It also knows about your aliases, lateral views and complex types and will include those. It will now automatically backtick any reserved words or exotic column names where needed to prevent any mistakes.</p> 1802<p><strong>Smart keyword completion</strong></p> 1803<p>The autocompleter suggests keywords based on where the cursor is positioned in the statement. Where possible it will even suggest more than one word at a time, like in the case of IF NOT EXISTS, no one likes to type too much right? In the parts where order matters but the keywords are optional, for instance after FROM tbl, it will list the keyword suggestions in the order they are expected with the first expected one on top. So after FROM tbl the WHERE keyword is listed above GROUP BY etc.</p> 1804<p><strong>Functions</strong></p> 1805<p>The improved autocompleter will now suggest functions, for each function suggestion an additional panel is added in the autocomplete dropdown showing the documentation and the signature of the function. The autocompleter know about the expected types for the arguments and will only suggest the columns or functions that match the argument at the cursor position in the argument list.</p> 1806<!-- raw HTML omitted --> 1807<p><strong>Sub-queries, correlated or not</strong></p> 1808<p>When editing subqueries it will only make suggestions within the scope of the subquery. For correlated subqueries the outside tables are also taken into account.</p> 1809<p><strong>Context popup</strong></p> 1810<p>Right click on any fragment of the queries (e.g. a table name) to gets all its metadata information. This is a handy shortcut to get more description or check what types of values are contained in the table or columns.</p> 1811<p>Itâs quite handy to be able to look at column samples while writing a query to see what type of values you can expect. Hue now has the ability to perform some operations on the sample data, you can now view distinct values as well as min and max values. Expect to see more operations in coming releases.</p> 1812<p><img src="https://cdn.gethue.com/uploads/2018/10/sample_context_operations.gif" alt="Sample column popup"></p> 1813<p><strong>Syntax checker</strong></p> 1814<p>A little red underline will display the incorrect syntax so that the query can be fixed before submitting. A right click offers suggestions.</p> 1815<p><img src="https://cdn.gethue.com/uploads/2018/01/syntax_checkerhigh.png" alt="Syntax checker"></p> 1816<p><img src="https://cdn.gethue.com/uploads/2018/01/checker_help.png" alt="Syntax checker"></p> 1817<p><strong>Advanced Settings</strong></p> 1818<p>The live autocompletion is fine-tuned for a better experience advanced settings an be accessed via <code>CTRL +</code> , (or on Mac <code>CMD + ,</code>) or clicking on the ‘?’ icon.</p> 1819<p>The autocompleter talks to the backend to get data for tables and databases etc and caches it to keep it quick. Clicking on the refresh icon in the left assist will clear the cache. This can be useful if a new table was created outside of Hue and is not yet showing-up (Hue will regularly clear his cache to automatically pick-up metadata changes done outside of Hue).</p> 1820<h3 id="sharing">Sharing</h3> 1821<p>Any query can be shared with permissions, as detailed in the <a href="/user/concept/">concepts</a>.</p> 1822<h3 id="assist">Assist</h3> 1823<p>The Datawarehouse ecosystem is getting more complete with the introduction of transactions. In practice, this means your tables can now support <code>Primary Keys</code>, <code>INSERTs</code>, <code>DELETEs</code> and <code>UPDATEs</code> as well as <code>Partition Keys</code>.</p> 1824<p>Here is a tutorial demoing how Hue's SQL Editor helps you quickly visualize and use these instructions via its <a href="/user/concept/">assists</a> and <a href="/user/querying/#autocomplete">autocomplete</a> components.</p> 1825<p><img src="https://cdn.gethue.com/uploads/2019/11/sql_column_pk.png" alt="Assist All Keys"></p> 1826<h4 id="primary-keys">Primary Keys</h4> 1827<p>Primary Keys shows up like Partition Keys with the lock icon:</p> 1828<p><img src="https://cdn.gethue.com/uploads/2019/11/sql_columns_assist_pks.png" alt="Assist Primary Keys"></p> 1829<p>Here is an example of SQL for using them:</p> 1830<pre><code>CREATE TABLE customer ( 1831 first_name string, 1832 last_name string,
1833 website string, 1834 PRIMARY KEY (first_name, last_name) DISABLE NOVALIDATE 1835); 1836</code></pre> 1837<p><a href="https://kudu.apache.org/">Apache Kudu</a> is supported as well:</p> 1838<pre><code>CREATE TABLE students ( 1839 id BIGINT, 1840 name STRING, 1841 PRIMARY KEY(id) 1842) 1843PARTITION BY HASH PARTITIONS 16 1844STORED AS KUDU 1845TBLPROPERTIES ('kudu.num_tablet_replicas' = '1') 1846; 1847</code></pre> 1848<h4 id="foreign-keys">Foreign Keys</h4> 1849<p>When a column value points to another column in another table. e.g. The head of the business unit must exist in the person table:</p> 1850<p><img src="https://cdn.gethue.com/uploads/2020/03/assist_foreign_keys_icons.png" alt="Assist Foreign Keys"></p> 1851<pre><code>CREATE TABLE person ( 1852 id INT NOT NULL, 1853 name STRING NOT NULL, 1854 age INT, 1855 creator STRING DEFAULT CURRENT_USER(), 1856 created_date DATE DEFAULT CURRENT_DATE(), 1857 1858 PRIMARY KEY (id) DISABLE NOVALIDATE 1859); 1860 1861CREATE TABLE business_unit ( 1862 id INT NOT NULL, 1863 head INT NOT NULL, 1864 creator STRING DEFAULT CURRENT_USER(), 1865 created_date DATE DEFAULT CURRENT_DATE(), 1866 1867 PRIMARY KEY (id) DISABLE NOVALIDATE, 1868 CONSTRAINT fk FOREIGN KEY (head) REFERENCES person(id) DISABLE NOVALIDATE 1869); 1870</code></pre> 1871<h4 id="partition-keys">Partition Keys</h4> 1872<p>Partitioning of the data is a key concept for optimizing the querying. Those special columns are also shown with a key icon:</p> 1873<p><img src="https://cdn.gethue.com/uploads/2019/11/sql_columns_assist_keys.png" alt="Assist Column Partition Keys"></p> 1874<p>Here is an example of SQL for using them:</p> 1875<pre><code>CREATE TABLE web_logs ( 1876 _version_ BIGINT, 1877 app STRING, 1878 bytes SMALLINT, 1879 city STRING, 1880 client_ip STRING, 1881 code TINYINT, 1882 country_code STRING, 1883 country_code3 STRING, 1884 country_name STRING, 1885 device_family STRING, 1886 extension STRING, 1887 latitude FLOAT, 1888 longitude FLOAT, 1889 `METHOD` STRING, 1890 os_family STRING, 1891 os_major STRING, 1892 protocol STRING, 1893 record STRING, 1894 referer STRING, 1895 region_code BIGINT, request STRING, 1896 subapp STRING, 1897 TIME STRING, 1898 url STRING, 1899 user_agent STRING, 1900 user_agent_family STRING, 1901 user_agent_major STRING, 1902 id STRING 1903) 1904PARTITIONED BY ( `date` STRING); 1905 1906INSERT INTO web_logs 1907PARTITION (`date`='2019-11-14') VALUES 1908(1480895575515725824,'metastore',1041,'Singapore','128.199.234.236',127,'SG','SGP','Singapore','Other',NULL,1.2930999994277954,103.85579681396484,'GET','Other',NULL,'HTTP/1.1',NULL,'-',0,'GET /metastore/table/default/sample_07 HTTP/1.1','table','2014-05-04T06:35:49Z','/metastore/table/default/sample_07','Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)','Other',NULL,'8836e6ce-9a21-449f-a372-9e57641389b3') 1909</code></pre> 1910<h4 id="nested-types">Nested Types</h4> 1911<p>Complex or Nested Types are handy for storing associated data close together. The assist lets you expand the tree of columns:</p> 1912<p><img src="https://cdn.gethue.com/uploads/2019/11/sql_columns_assist_nested_types.png" alt="Assist Nested Types"></p> 1913<p>Here is an example of SQL for using them:</p> 1914<pre><code>CREATE TABLE subscribers ( 1915 id INT, 1916 name STRING, 1917 email_preferences STRUCT<email_format:STRING,frequency:STRING,categories:STRUCT<promos:BOOLEAN,surveys:BOOLEAN>>, 1918 addresses MAP<STRING,STRUCT<street_1:STRING,street_2:STRING,city:STRING,state:STRING,zip_code:STRING>>, 1919 orders ARRAY<STRUCT<order_id:STRING,order_date:STRING,items:ARRAY<STRUCT<product_id:INT,sku:STRING,name:STRING,price:DOUBLE,qty:INT>>>> 1920) 1921STORED AS PARQUET 1922</code></pre> 1923<h4 id="views">Views</h4> 1924<p>It can be sometimes confusing to not recognize that a table is instead a view. Views are shown with this little eye icon:</p> 1925<p><img src="https://cdn.gethue.com/uploads/2019/11/sql_assist_view_icon.png" alt="Assist Nested Types"></p> 1926<p>Here is an example of SQL for using them:</p> 1927<pre><code>CREATE VIEW web_logs_november AS 1928SELECT * FROM web_logs 1929WHERE `date` BETWEEN '2019-11-01' AND '2019-12-01' 1930</code></pre> 1931<h4 id="transactional-operations">Transactional Operations</h4> 1932<p>Transactional tables now support these SQL instructions to update the data.</p> 1933<h5 id="inserts">Inserts</h5> 1934<p>Here is how to add some data into a table. Previously, he was only possible to do this via LOADING some files.</p> 1935<pre><code>INSERT INTO TABLE customer 1936VALUES 1937 ('Elli', 'SQL', 'gethue.com'), 1938 ('John', 'SELECT', 'docs.gethue.com') 1939; 1940</code></pre> 1941<h5 id="deletes">Deletes</h5> 1942<p>Deletion of rows of data:</p> 1943<pre><code>DELETE FROM customer 1944WHERE first_name = 'John'; 1945</code></pre> 1946<h5 id="updates">
1946Updates</h5> 1947<p>How to update the value of some columns in certain rows:</p> 1948<pre><code>UPDATE customer 1949SET website = 'helm.gethue.com' 1950WHERE first_name = 'Elli'; 1951</code></pre> 1952<h5 id="language-reference">Language reference</h5> 1953<p>You can find the Language Reference in the right assist panel. The right panel itself has a new look with icons on the left hand side and can be minimised by clicking the active icon.</p> 1954<p>The filter input on top will only filter on the topic titles in this initial version. Below is an example on how to find documentation about joins in select statements.</p> 1955<p><img src="https://cdn.gethue.com/uploads/2018/10/impala_lang_ref_joins.gif" alt="Language Reference Panel"></p> 1956<p>While editing a statement thereâs a quicker way to find the language reference for the current statement type, just right-click the first word and the reference appears in a popover below:</p> 1957<p><img src="https://cdn.gethue.com/uploads/2018/10/impala_lang_ref_context.png" alt="Language Reference context"></p> 1958<h3 id="variables">Variables</h3> 1959<p>Variables are used to easily configure parameters in a query. They are ideal for saving reports that can be shared or executed repetitively:</p> 1960<p><strong>Single Valued</strong></p> 1961<pre><code>select * from web_logs where country_code = "${country_code}" 1962</code></pre> 1963<p><img src="https://cdn.gethue.com/uploads/2017/10/var_defaults.png" alt="Single valued variable"></p> 1964<p><strong>The variable can have a default value</strong></p> 1965<pre><code>select * from web_logs where country_code = "${country_code=US}" 1966</code></pre> 1967<p><strong>Multi Valued</strong></p> 1968<pre><code>select * from web_logs where country_code = "${country_code=CA, FR, US}" 1969</code></pre> 1970<p><strong>In addition, the displayed text for multi valued variables can be changed</strong></p> 1971<pre><code>select * from web_logs where country_code = "${country_code=CA(Canada), FR(France), US(United States)}" 1972</code></pre> 1973<p><img src="https://cdn.gethue.com/uploads/2018/04/variables_multi.png" alt="Multi valued variables"></p> 1974<p><strong>For values that are not textual, omit the quotes.</strong></p> 1975<pre><code>select * from boolean_table where boolean_column = ${boolean_column} 1976</code></pre> 1977<h3 id="charting">Charting</h3> 1978<p>These visualizations are convenient for plotting chronological data or when subsets of rows have the same attribute: they will be stacked together.</p> 1979<ul> 1980<li>Pie</li> 1981<li>Bar/Line with pivot</li> 1982<li>Timeline</li> 1983<li>Scattered plot</li> 1984<li>Maps (Marker and Gradient)</li> 1985</ul> 1986<p><img src="https://cdn.gethue.com/uploads/2019/04/editor_charting.png" alt="Charts"></p> 1987<h3 id="query-troubleshooting">Query troubleshooting</h3> 1988<h4 id="pre-query-execution">Pre-query execution</h4> 1989<p><strong>Popular values</strong></p> 1990<p>The autocompleter will suggest popular tables, columns, filters, joins, group by, order by etc. based on metadata from Navigator Optimizer. A new âPopularâ tab has been added to the autocomplete result dropdown which will be shown when there are popular suggestions available.</p> 1991<p>This is particularly useful for doing joins on unknown datasets or getting the most interesting columns of tables with hundreds of them.</p> 1992<p><img src="https://cdn.gethue.com/uploads/2017/07/hue_4_query_joins.png" alt="Popular joins suggestion"> 1993<img src="https://cdn.gethue.com/uploads/2017/07/hue_4_popular_filter_agg.png" alt="Popular columns suggestion"></p> 1994<p><strong>Risk alerts</strong></p> 1995<p>While editing, Hue will run your queries through Navigator Optimizer in the background to identify potential risks that could affect the performance of your query. If a risk is identified an exclamation mark is shown above the query editor and suggestions on how to improve it is displayed in the lower part of the right assistant panel.</p> 1996<p><img src="https://cdn.gethue.com/uploads/2017/07/hue_4_risk_6.gif" alt="Query Risk alerts"></p> 1997<h4 id="during-execution">During execution</h4> 1998<p>The <a href="/user/browsing/#sql-queries">Query Browser</a> details the plan of the query and the bottle necks. When detected, “Health” risks are listed with suggestions on how to fix them.</p> 1999<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-11.40.24-AM.png" alt="Pretty Query Profile"></p> 2000<h4 id="tutorial">Tutorial</h4> 2001<p>After finding data in the Catalog and using the Query Assistant, end users might wonder why their queries are taking a lot of time to execute. Build up on top of the Impala profiler, this new feature educates them and surface up more information so that they can be more productive by themselves. Here is a scenario that showcases the flow:</p> 2002<p><strong>Execution Timeline</strong></p> 2003<p>To give you a feel for the new features, weâll execute a few queries.</p> 2004<pre><code>SELECT * 2005FROM 2006 transactions1g s07 left JOIN transactions1g s08 2007ON ( s07.field_1 = s08.field_1) limit 100 2008</code></pre> 2009<p>
2009transactions1g is a 1GB table and the self join with no predicates will force a network transfer of the whole table.</p> 2010<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-06-at-4.08.01-PM.png" alt="Impala Profile"></p> 2011<p>Looking at the profile, you can see a number on the top right of each node that represent its IO and CPU time. Thereâs also a timeline that gives an estimated representation of when that node was processed during execution. The dark blue color is the CPU time, while the lighter blue is the network or disk IO time. In this example, we can see that the hash join ran for 2.5s. The exchange node, which does the network transfer between 2 hosts, was the most expensive node at 7.2s.</p> 2012<p><strong>Detail pane</strong></p> 2013<p>On the right hand side, there is now a pane that is closed by default. To open or close press on the header of the pane. There, you will find a list of all the nodes sorted by execution time, which makes it easier to navigate larger execution graphs. This list is clickable and will navigate to the appropriate node.</p> 2014<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-06-at-4.12.38-PM.png" alt="Impala Profile"></p> 2015<p><strong>Events</strong></p> 2016<p>Pressing on the exchange node, we find the execution timeline with a bit more detail.</p> 2017<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-06-at-4.13.40-PM.png" alt="Impala Profile"></p> 2018<p>We see that the IO was the most significant portion of the exchange.</p> 2019<p><strong>Statistics by host</strong></p> 2020<p>The detail pane also contains detailed statistics aggregated per host per node such as memory consumption and network transfer speed.</p> 2021<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-06-at-4.16.11-PM.png" alt="Impala Profile"></p> 2022<p><strong>Risks</strong></p> 2023<p>In the detail pane, for each node, you will find a section titled risks. This section will contain hints on how to improve performance for this operator. Currently, this is not enabled by default. To enable it, go to your Hue ini file and enable this flag:</p> 2024<pre><code>[notebook] 2025enable_query_analysis=true 2026</code></pre> 2027<p><strong>CodeGen</strong></p> 2028<p>Letâs look at a few queries and some of the risks that can be identified.</p> 2029<pre><code>SELECT s07.description, s07.salary, s08.salary, 2030 s08.salary - s07.salary 2031FROM 2032 sample_07 s07 left outer JOIN sample_08 s08 2033ON ( s07.code = s08.code) 2034where s07.salary > 100000 2035</code></pre> 2036<p>sample_07 & sample_08 are small sample tables that come with Hue.</p> 2037<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-9.35.23-AM.png" alt="Impala Profile"></p> 2038<p>Looking at the graph, the timelines are mostly empty. If we open one of the nodes we see that all the time is taken by âCodeGenâ.</p> 2039<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-9.40.50-AM.png" alt="Impala Profile"></p> 2040<p>Impala compiles SQL requests to native code to execute each node in the graph. On queries with large table this gives a large performance boost. On smaller tables, we can see that CodeGen is the main contributor to execution time. Normally, Impala disables CodeGen with tables of small sizes, but Impala doesnât know itâs a small table as is pointed out in the risks section by the statement âStatistics missingâ. Two solutions are available here:</p> 2041<p>Adding the missing statistics. One way to do this is to execute the following command:</p> 2042<pre><code>compute stats sample_07; 2043compute stats sample_08; 2044</code></pre> 2045<p>This is usually the right thing to do, but on larger tables it can be quite expensive.</p> 2046<p>Disable codegen for the query via:</p> 2047<pre><code>set DISABLE_CODEGEN=true 2048</code></pre> 2049<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-9.52.37-AM.png" alt="Impala Profile"></p> 2050<p>After rerunning the query, we see that CodeGen is now gone.</p> 2051<p><strong>Join Order</strong></p> 2052<p>If we open the join node, thereâs a warning for wrong join order.</p> 2053<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-4.50.54-PM.png" alt="Impala Profile"></p> 2054<p>Impala prefers having the table with the larger size on the right hand side of the graph, but in this case the reverse is true. Normally, Impala would optimize this automatically, but we saw that the statistics were missing for the tables being joined. There a few ways we could fix this:</p> 2055<ul> 2056<li> 2057<p>Add the missing statistics as described earlier.</p> 2058</li> 2059<li> 2060<p>Rewrite the query the change the join order:</p> 2061<pre><code>SELECT s07.description, s07.salary, s08.salary, 2062 s08.salary - s07.salary 2063FROM 2064 sample_08 s08 left outer JOIN sample_07 s07 2065ON ( s07.code = s08.code) 2066where s07.salary > 100000 2067</code></pre></li> 2068</ul> 2069<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-9.57.14-AM.png" alt="Impala Profile"></p> 2070<p>The warning is gone and the execution time for the join is down.<
2070/p> 2071<p><strong>Spilling</strong></p> 2072<p>Impala will execute all of its operators in memory if enough is available. If the execution does not all fit in memory, Impala will use the available disk to store its data temporarily. To see this in action, weâll use the same query as before, but weâll set a memory limit to trigger spilling:</p> 2073<pre><code>set MEM_LIMIT=1g; 2074select * 2075FROM 2076 transactions1g s07 left JOIN transactions1g s08 2077ON ( s07.field_1 = s08.field_1); 2078</code></pre> 2079<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-11.40.24-AM.png" alt="Impala Profile"></p> 2080<p>Looking at the join node, we can see that thereâs an entry in the risk section about a spilled partition. Typically, the join only has CPU time, but in this case it also has IO time due to the spill.</p> 2081<p><strong>Kudu Filtering</strong></p> 2082<p>Kudu is one of the supported storage backends for Impala. While Impala stand alone can query a variety of file data formats, Impala on Kudu allows fast updates and inserts on your data, and also is a better choice if small files are involved. When using Impala on Kudu, Impala will push down some of the operations to Kudu to reduce the data transfer between the two.</p> 2083<p>However, Kudu does not support all the operators that Impala support. For example, at the time of writing, Impala support the âlikeâ operator, but Kudu does not. In those cases, all the data that cannot be natively filtered in Kudu is transferred to Impala where it will be filtered. Letâs look at a behavior difference between the two.</p> 2084<pre><code>SELECT * FROM transactions1g_kudu s07 left JOIN transactions1g_kudu s08 on s07.field_1 = s08.field_1 2085where s07.field_5 LIKE '2000-01%'; 2086</code></pre> 2087<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-5.00.59-PM.png" alt="Impala Profile"></p> 2088<p>When we look at the graph, we see that on the Kudu node we have both IO, which represent the time spent in Kudu, and CPU, which represent the time spent in Impala, for a total of 2.1s. In the risk section, we can also find a warning that Kudu could not evaluate the predicate.</p> 2089<pre><code>SELECT * FROM transactions1g_kudu s07 left JOIN transactions1g_kudu s08 on s07.field_1 = s08.field_1 2090where s07.field_5 <= '2000-01-31' and s07.field_5 >= '2000-01-01'; 2091</code></pre> 2092<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-4.02.33-PM.png" alt="Impala Profile"></p> 2093<p>When we look a the graph, we see that on the Kudu node now mostly has IO for a total time 727ms.</p> 2094<p><strong>Others</strong></p> 2095<p>You might also have queries where the nodes have short execution time, but the total duration time is long. Using the same query, we see all the nodes have sub 10ms execution time, but the query execution was 7.9s.</p> 2096<p><img src="https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-10.56.07-AM.png" alt="Impala Profile"></p> 2097<p>Looking at the global timeline, we see that the planning phase took 3.8s with most of the time in metadata load. When Impala doesnât have metadata about a table, which can happen after a user executes:</p> 2098<pre><code>invalidate metadata; 2099</code></pre> 2100<p>Impala has to refetch the metadata from the metastore. Furthermore, we see that the second most expensive item at 4.1s is first row fetched. This is the time it took the client, Hue in this case, to fetch the results. While both of these events are not things that a user can change, itâs good to see where the time is spent.</p> 2101<h4 id="post-query-execution">Post-query execution</h4> 2102<p>A new experimental panel when enabled can offer post risk analysis and recommendation on how to tweak the query for better speed.</p> 2103<h3 id="modes">Modes</h3> 2104<h4 id="presentation">Presentation</h4> 2105<p>Turns a list of semi-colon separated queries into an interactive presentation by clicking on the ‘Dashboard’ icon. It is great for doing presentations with a scenario and live results to prove a point or executing reports containting a suite of sequential queries in one click.</p> 2106<p><img src="https://cdn.gethue.com/uploads/2020/02/editor_presentation_mode.png" alt="Editor Presentation Mode"></p> 2107<h4 id="dark">Dark</h4> 2108<p>Initially this mode is limited to the actual editor area and weâre considering extending this to cover all of Hue.</p> 2109<p><img src="https://cdn.gethue.com/uploads/2018/10/editor_dark_mode.png" alt="Editor Dark Mode"></p> 2110<p>To toggle the dark mode you can either press <code>Ctrl-Alt-T</code> or <code>Command-Option-T</code> on Mac while the editor has focus. Alternatively you can control this through the settings menu which is shown by pressing <code>Ctrl-</code>, or <code>Command-</code>, on Mac.</p> 2111<h2 id="dashboard">Dashboard</h2> 2112<p>Dashboards provide an interactive way to query indexed data quickly and easily. No programming is required and the analysis is done by drag & drops and clicks.</p> 2113<p><img src="https://cdn.gethue.com/uploads/2015/08/search-full-mode.png" alt="Search Full"></p> 2114<p>Widgets are interconnected together. This is great for exploring new datasets or monitoring without having to type.</p> 2115<p><img src="https://cdn.gethue.com/uploads/2018/08/dashboard_layout_dnd.gif" alt="Analytics dimensions"></p> 2116<p>The best supported engine is Apache Solr, then support for SQL databases is getting better. To help add more SQL support, feel free to check the <a href="/developer/development/#connectors">dashboard connector</a> section.</p> 2117<p>These tutorials showcase the capabilities:</p> 2118<ul> 2119<li>The top search bar offers a <a href="http://gethue.com/intuitively-discovering-and-exploring-a-wine-dataset-with-the-dynamic-dashboards/">full autocomplete</a> on all the values of the index</li> 2120<li>Seeing <a href="http://gethue.com/build-a-real-time-analytic-dashboard-with-solr-search-and-spark-streaming/">real time data</a></li> 2121<li>
2121Comprehensive demo of <a href="http://gethue.com/bay-area-bikeshare-data-analysis-with-search-and-spark-notebook/">BikeShare data visualization post</a></li> 2122</ul> 2123<h3 id="analytics-facets">Analytics facets</h3> 2124<p>Drill down the dimensions of the datasets and apply aggregates functions on top of it:</p> 2125<p><img src="https://cdn.gethue.com/uploads/2018/08/dashboard_layout_dimensions.gif" alt="Analytics dimensions"></p> 2126<p>Some facets can be nested:</p> 2127<p><img src="https://cdn.gethue.com/uploads/2015/08/search-nested-facet-1024x304.png" alt="Nested Analytics facets"> 2128<img src="https://cdn.gethue.com/uploads/2015/08/search-hit-widget.png" alt="Nested Analytics Counts"></p> 2129<h3 id="autocomplete-1">Autocomplete</h3> 2130<p>The top bar support faceted and free word text search, with autocompletion.</p> 2131<p><img src="https://cdn.gethue.com/uploads/2018/01/dashboard_autocomplete.png" alt="Search Autocomplete"></p> 2132<h3 id="marker-map">Marker Map</h3> 2133<p>Points close to each other are grouped together and will expand when zooming-in. A Yelp-like search f
2133iltering experience can also be created by checking the box.</p> 2134<p><img src="https://cdn.gethue.com/uploads/2015/08/search-marker-map.png" alt="Marker Map"></p> 2135<h3 id="edit-records">Edit records</h3> 2136<p>Indexed records can be directly edited in the Grid or HTML widgets by admins.</p> 2137<h3 id="links">Links</h3> 2138<p>Links to the original documents can also be inserted. Add to the record a field named âlink-metaâ that contains some json describing the URL or address of a table or file that can be open in the HBase Browser, Metastore App or File Browser:</p> 2139<p>Any link</p> 2140<pre><code>{'type': 'link', 'link': 'gethue.com'} 2141</code></pre> 2142<p>HBase Browser</p> 2143<pre><code>{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527'} 2144{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527', 'fam': 'f1'} 2145{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527', 'fam': 'f1', 'col': 'c1'} 2146</code></pre> 2147<p>File Browser</p> 2148<pre><code>{'type': 'hdfs', 'path': '/data/hue/file.txt'} 2149</code></pre> 2150<p>Table Catalog</p> 2151<pre><code>{'type': 'hive', 'database': 'default', 'table': 'sample_07'} 2152</code></pre> 2153<p><img src="https://cdn.gethue.com/uploads/2015/08/search-link-1024x630.png" alt="Data Links"></p> 2154<h3 id="saved-queries">Saved queries</h3> 2155<p>Current selected facets and filters, query strings can be saved with a name within the dashboard. These are useful for defining âcohortsâ or pre-selection of records and quickly reloading them.</p> 2156<p><img src="https://cdn.gethue.com/uploads/2015/08/search-query-def-1024x507.png" alt="Rolling time"></p> 2157<h3 id="fixed-or-rolling-window">âFixedâ or ârollingâ window</h3> 2158<p>Real time indexing can now shine with the rolling window filter and the automatic refresh of the dashboard every N seconds. See it in action in the real time Twitter indexing with Spark streaming post.</p> 2159<p><img src="https://cdn.gethue.com/uploads/2015/08/search-fixed-time.png" alt="Fixed time"></p> 2160<h3 id="more-like-this">‘More like this’</h3> 2161<p>This feature lets you selected fields you would like to use to find similar records. This is a great way to find similar issues, customers, people… with regard to a list of attributes.</p> 2162<p><img src="https://cdn.gethue.com/uploads/2018/01/solr_more_like_this.png" alt="More like this"></p> 2163<h2 id="notebook">Notebook</h2> 2164<p>The goal of Notebooks is to quickly experiment with small programming snippets (with in particular Spark) and do interactive demos. Its goal is to stay lightweight with regards to other notebook or programming systems.</p> 2165<p>The main advantage is to be able to add snippets of different dialects (e.g. PySpark, Hive SQL…) into a single page:</p> 2166<p><img src="https://cdn.gethue.com/uploads/2015/10/notebook-october.png" alt="Notebook mode"></p> 2167<p>Any configured language of the Editor will be available as a dialect. Each snippet has a code editor, with autocomplete, syntax highlighting and other feature like shortcut links to HDFS paths and Hive tables.</p> 2168<p><img src="https://cdn.gethue.com/uploads/2015/08/notebook.png" alt="Notebook Screen"></p> 2169<p>Example of SparkR shell with inline plot</p> 2170<p><img src="https://cdn.gethue.com/uploads/2015/08/spark-r-snippet.png" alt="Notebook r snippet"></p> 2171<p>All the spark-submit, spark-shell, pyspark, sparkR properties of jobs & shells can be added to the sessions of a Notebook. This will for example let you add files, modules and tweak the memory and number of executors.</p> 2172<p><img src="https://cdn.gethue.com/uploads/2015/08/notebook-sessions.png" alt="Notebook sessions"></p> 2173<h3 id="spark">Spark</h3> 2174<p>Hue relies on <a href="https://livy.incubator.apache.org/">Livy</a> for the interactive Scala, Python, SparkSQL and R snippets.</p> 2175<p>Livy is an open source REST interface for interacting with Apache Spark from anywhere. It got initially developed in the Hue project but got a lot of traction and was moved to its own project on livy.io.</p> 2176<p>Make sure that the Notebook and interpreters <a href="/administrator/configuration/connectors/#apache-spark">configured</a>.</p> 2177<h4 id="livy">Livy</h4> 2178<p>Starting the Livy REST server is detailed on the <a href="http://livy.incubator.apache.org/get-started/">get started</a> page.</p> 2179<p><strong>Executing some Spark</strong></p> 2180<p>As the REST server is running, we can communicate with it. We are on the same machine so will use âlocalhostâ as the address of Livy.</p> 2181<p>Letâs list our open sessions</p> 2182<pre><code>curl localhost:8998/sessions 2183 2184{"from":0,"total":0,"sessions":[]} 2185</code></pre> 2186<p><strong>Note</strong> You can use</p> 2187<pre><code>| python -m json.tool 2188</code></pre> 2189<p>at the end of the command to prettify the output, e.g.:</p> 2190<pre><code>curl localhost:8998/sessions/0 | python -m json.tool 2191</code></pre> 2192<p>There is zero session. We create an interactive PySpark session</p> 2193<pre><code>curl -X POST --data '{"kind": "pyspark"}' -H "Content-Type: application/json" localhost:8998/sessions 2194 2195{"id":0,"state":"starting","kind":"pyspark","log":[]} 2196</code></pre> 2197<p>Sessions ids are incrementing numbers starting from 0. We can then reference the session later by its id.</p> 2198<p>We check the status of the session until its state becomes idle: it means it is ready to be execute snippet of PySpark:</p> 2199<pre><code>curl localhost:8998/sessions/0 | python -m json.tool 2200 2201 2202% Total % Received % Xferd Average Speed Time Time Time Current 2203 2204 Dload Upload Total Spent Left Speed 2205 2206100 1185 0 1185 0 0 72712 0 --:--:-- --:--:-- --:--:-- 79000 2207 2208{ 2209 2210 "id": 5, 2211 2212 "kind": "pyspark", 2213 2214 "log": [ 2215 2216 "15/09/03 17:44:14 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.", 2217 2218 "15/09/03 17:44:14 INFO ui.SparkUI: Started SparkUI at http://172.21.2.198:4040", 2219 2220 "15/09/03 17:44:14 INFO spark.SparkContext: Added JAR file:/home/romain/projects/hue/apps/spark/java-lib/livy-assembly.jar at http://172.21.2.198:33590/jars/livy-assembly.jar with timestamp 1441327454666", 2221 2222 "15/09/03 17:44:14 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.", 2223 2224 "15/09/03 17:44:14 INFO executor.Executor: Starting executor ID driver on host localhost", 2225 2226 "15/09/03 17:44:14 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54584.", 2227 2228 "15/09/03 17:44:14 INFO netty.NettyBlockTransferService: Server created on 54584", 2229 2230 "15/09/03 17:44:14 INFO storage.BlockManagerMaster: Trying to register BlockManager", 2231 2232 "15/09/03 17:44:14 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:54584 with 530.3 MB RAM, BlockManagerId(driver, localhost, 54584)", 2233 2234 "15/09/03 17:44:15 INFO storage.BlockManagerMaster: Registered BlockManager" 2235 2236 ], 2237 2238 "state": "idle" 2239 2240} 2241</code></pre> 2242<p><img src="https://cdn.gethue.com/uploads/2015/09/20150818_scalabythebay.024.png" alt="Livy Architecture sessions"></p> 2243<p><strong>Session properties</strong></p> 2244<p>All the properties supported by spark shells like the number of executors, the memory, etc can be changed at session creation. Their format is the same as when typing spark-shell -h</p> 2245<pre><code>curl -X POST --data '{"kind": "pyspark", "numExecutors": "3", "executorMemory": "2G"}' -H "Content-Type: application/json" localhost:8998/sessions 2246{"id":0,"state":"starting","kind":"pyspark","numExecutors":"3","executorMemory":"2G","log":[]} 2247</code></pre> 2248<p><strong>Executing statements</strong></p> 2249<p>In YARN mode, Livy creates a remote Spark Shell in the cluster that can be accessed easily with REST</p> 2250<p>When the session state is idle, it means it is ready to accept statements! Lets compute 1 + 1</p> 2251<pre><code>curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' 2252 2253{"id":0,"state":"running","output":null} 2254</code></pre> 2255<p>We check the result of statement 0 when its state is available</p> 2256<pre><code>curl localhost:8998/sessions/0/statements/0 2257 2258{"id":0,"state":"available","output":{"status":"ok","execution_count":0,"data":{"text/plain":"2"}}} 2259</code></pre> 2260<p><strong>Note</strong> If the statement is taking less than a few milliseconds, Livy returns the response directly in the response of the POST command.</p> 2261<p>Statements are incrementing and all share the same context, so we can have a sequences</p> 2262<pre><code>curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"a = 10"}' 2263 2264{"id":1,"state":"available","output":{"status":"ok","execution_count":1,"data":{"text/plain":""}}} 2265</code></pre> 2266<p>Spanning multiple statements</p> 2267<pre><code>curl localhost:8998/sessions/5/statements -X POST -H 'Content-Type: application/json' -d '{"code":"a + 1"}' 2268 2269{"id":2,"state":"available","output":{"status":"ok","execution_count":2,"data":{"text/plain":"11"}}} 2270</code></pre> 2271<p>Letâs close the session to free up the cluster. Note that Livy will automatically inactive idle sessions after 1 hour (configurable).</p> 2272<pre><code>curl localhost:8998/sessions/0 -X DELETE 2273 2274{"msg":"deleted"} 2275</code></pre> 2276<h4 id="tutorial-sharing-rdds">Tutorial: Sharing RDDs</h4> 2277<p>This section shows how to share Spark RDDs and contexts. Livy offers remote Spark sessions to users. They usually have one each (or one by Notebook):</p> 2278<pre><code># Client 1 2279curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' 2280# Client 2 2281curl localhost:8998/sessions/1/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' 2282# Client 3 2283curl localhost:8998/sessions/2/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' 2284livy_shared_contexts2 2285</code></pre> 2286<p><img src="https://cdn.gethue.com/uploads/2015/10/livy_shared_contexts2.png" alt="Livy shared context"></p> 2287<h5 id="-and-so-sharing-rdds">… and so sharing RDDs</h5> 2288<p>If the users were pointing to the same session, they would interact with the same Spark context. This context would itself manages several RDDs. Users simply need to use the same session id, e.g. 0, and issue commands there:</p> 2289<pre><code># Client 1 2290curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' 2291 2292# Client 2 2293curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' 2294 2295# Client 3 2296curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' 2297</code></pre> 2298<p><img src="https://cdn.gethue.com/uploads/2015/10/livy_multi_rdds2.png" alt="Livy multi rdds"></p> 2299<h5 id="accessing-them-from-anywhere">…Accessing them from anywhere</h5> 2300<p>
2300Now we can even make it more sophisticated while keeping it simple. Imagine we want to simulate a shared in memory key/value store. One user can start a named RDD on a remote Livy PySpark session and anybody could access it.</p> 2301<p><img src="https://cdn.gethue.com/uploads/2015/10/livy_shared_rdds_anywhere2.png" alt="Livy anywhere rdds"></p> 2302<p>To make it prettier, we can wrap it in a few lines of Python and call it ShareableRdd. Then users can directly connect to the session and set or retrieve values.</p> 2303<pre><code>class ShareableRdd(): 2304 2305def __init__(self): 2306 self.data = sc.parallelize([]) 2307 2308def get(self, key): 2309 return self.data.filter(lambda row: row[0] == key).take(1) 2310 2311def set(self, key, value): 2312 new_key = sc.parallelize([[key, value]]) 2313 self.data = self.data.union(new_key) 2314</code></pre> 2315<p>set() adds a value to the shared RDD, while get() retrieves it.</p> 2316<pre><code>srdd = ShareableRdd() 2317 2318srdd.set('ak', 'Alaska') 2319srdd.set('ca', 'California') 2320 2321srdd.get('ak') 2322</code></pre> 2323<p>If using the REST Api directly someone can access it with just these commands:</p> 2324<pre><code>curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"srdd.get(\"ak\")"}' 2325{"id":3,"state":"running","output":null} 2326 2327curl localhost:8998/sessions/0/statements/3 2328{"id":3,"state":"available","output":{"status":"ok","execution_count":3,"data":{"text/plain":"[['ak', 'Alaska']]"}}} 2329</code></pre> 2330<p>We can even get prettier data back, directly in json format by adding the %json magic keyword:</p> 2331<pre><code>curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"data = srdd.get(\"ak\")\n%json data"}' 2332{"id":4,"state":"running","output":null} 2333 2334curl localhost:8998/sessions/0/statements/4 2335{"id":4,"state":"available","output":{"status":"ok","execution_count":2,"data":{"application/json":[["ak","Alaska"]]}}} 2336</code></pre> 2337<h5 id="even-from-any-languages">Even from any languages</h5> 2338<p>As Livy is providing a simple REST Api, we can quickly implement a little wrapper around it to offer the shared RDD functionality in any languages. Letâs do it with regular Python:</p> 2339<pre><code>pip install requests 2340python 2341</code></pre> 2342<p>Then in the Python shell just declare the wrapper:</p> 2343<pre><code>import requests 2344import json 2345 2346class SharedRdd(): 2347 """ 2348 Perform REST calls to a remote PySpark shell containing a Shared named RDD. 2349 """ 2350 def __init__(self, session_url, name): 2351 self.session_url = session_url 2352 self.name = name 2353 2354 def get(self, key): 2355 return self._curl('%(rdd)s.get("%(key)s")' % {'rdd': self.name, 'key': key}) 2356 2357 def set(self, key, value): 2358 return self._curl('%(rdd)s.set("%(key)s", "%(value)s")' % {'rdd': self.name, 'key': key, 'value': value}) 2359 2360 def _curl(self, code): 2361 statements_url = self.session_url + '/statements' 2362 data = {'code': code} 2363 r = requests.post(statements_url, data=json.dumps(data), headers={'Content-Type': 'application/json'}) 2364 resp = r.json() 2365 statement_id = str(resp['id']) 2366 while resp['state'] == 'running': 2367 r = requests.get(statements_url + '/' + statement_id) 2368 resp = r.json() 2369 return r.json()['data'] 2370</code></pre> 2371<p>Instantiate it and make it point to a live session that contains a ShareableRdd:</p> 2372<pre><code>states = SharedRdd('http://localhost:8998/sessions/0', 'states') 2373</code></pre> 2374<p>And just interact with the RDD transparently:</p> 2375<pre><code>states.get('ak') 2376states.set('hi', 'Hawaii') 2377</code></pre> 2378<h3 id="others">Others</h3> 2379<p><strong>Apache Pig</strong> 2380Type <a href="https://pig.apache.org/">Apache Pig</a> latin instructions to load/merge data to perform ETL or Analytics.</p> 2381<p><strong>Apache Sqoop</strong> 2382Run an <a href="/user/browsing/#relational-databases">SQL import</a> from a traditional relational database via an <a href="https://sqoop.apache.org/">Apache Sqoop</a> command.</p> 2383 2384 2385<footer class=" footline" > 2386 2387</footer> 2388 2389 2390 </div> 2391 2392 2393 </div> 2394 2395 <div id="navigation"> 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144
3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 </div> 3537 3538 </section> 3539 3540 <div style="left: -1000px; overflow: scroll; position: absolute; top: -1000px; border: none; box-sizing: content-box; height: 200px; margin: 0px; padding: 0px; width: 200px;"> 3541 <div style="border: none; box-sizing: content-box; height: 200px; margin: 0px; padding: 0px; width: 200px;"></div> 3542 </div> 3543
3543<script src="/js/clipboard.min.js"></script>
3543 3544
3544<script src="/js/perfect-scrollbar.min.js"></script>
3544 3545
3545<script src="/js/perfect-scrollbar.jquery.min.js"></script>
3545 3546
3546<script src="/js/jquery.sticky.js"></script>
3546 3547
3547<script src="/js/featherlight.min.js"></script>
3547 3548
3548<script src="/js/html5shiv-printshiv.min.js"></script>
3548 3549
3549<script src="/js/highlight.pack.js"></script>
3549 3550
3550<script>hljs.initHighlightingOnLoad();</script>
3550 3551
3551<script src="/js/modernizr.custom.71422.js"></script>
3551 3552
3552<script src="/js/learn.js"></script>
3552 3553
3553<script src="/js/hugo-learn.js"></script>
3553 3554 3555 <link href="/mermaid/mermaid.css" type="text/css" rel="stylesheet" /> 3556
3556<script src="/mermaid/mermaid.js"></script>
3556 3557
3557<script> 3558 mermaid.initialize({ startOnLoad: true }); 3559 </script>
vendor: 69 bytes, lines 3559-3561
3559 3560 3561<script async src="https://www.googletagmanager.com/gtag/js?id=
3561UA-37637545-1
vendor: 12 bytes, line 3561
3561"></script>
3562<script> 3563 window.dataLayer = window.dataLayer || []; 3564 function gtag(){dataLayer.push(arguments);} 3565 gtag('js', new Date()); 3566 3567 gtag('config', '${ conf.GTAG_ID.get()}', { 3568 3569 send_page_view: false, 3570 page_location: 'redacted', 3571 page_referrer: 'redacted', 3572 allow_google_signals: false, 3573 cookie_expires: 0 3574 }); 3575</script>
3575 3576
3576<script type="module" src="https://static.cloudflareinsights.com/beacon.min.js/v31edd6df95cf4e85bb4c19e7a9bdbcba1788362987495" integrity="sha512-iIg7k2xntmwu6/uSb5tpc/hySgZc4eoL31yB29W6tJFo2akwjPWcEqnCEdJvGexCL0KEQwVYv5BlowfhVz26hg==" data-cf-beacon='{"version":"2024.11.0","token":"9a293470f3934fccb678b763188cb19a","r":1,"spa":2}' crossorigin="anonymous"></script>
3576 3577</body> 3578</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.