|!Punctuation|!Location|!Function|\n|{{{@@...@@}}}|surrounding text|@@highlighted text@@|\n|{{{//...//}}}|surrounding text|//italicized text//|\n|{{{==...==}}}|surrounding text|==strikethrough text==|\n|{{{''...''}}}|surrounding text|''boldfaced text''|\n|{{{__...__}}}|surrounding text|__underlined text__|\n|{{{[[text|url]]}}}|around text/url pair|[[text|http://gri.gallaudet.edu/]] link to url|\n|{{{ {...} }}}|''__tripled__'' surrounding text|{{{in-line literal text}}}|\n|{{{ {...} }}}|''__tripled__'' surrounding ''__lines__''|literal block|\n|{{{<<<}}}|surrounding ''__lines__''|blockquotes|\n|{{{!}}}|at start of line|subheading|\n|{{{|...|...|}}}|line sectioned by vertical bars|table row|\n|{{{!}}}|in a table|!table heading|\n|{{{----}}}|alone on line|horizontal rule|\n|{{{*}}}|at start of line|bulleted list item|\n|{{{#}}}|at start of line|numbered list item|\nsource: Kevin Cole, January 2007
TTiddlyWiki uses Wiki style markup, a way of lightly "tagging" plain text so it can be transformed into HTML. Edit this Tiddler to see samples.\n\n! Header Samples\n!Header 1\n!!Header 2\n!!!Header 3\n!!!!Header 4\n!!!!!Header 5\n\n! Unordered Lists:\n* Lists are where it's at\n* Just use an asterisk and you're set\n** To nest lists just add more asterisks...\n***...like this\n* The circle makes a great bullet because once you've printed a list you can mark off completed items\n* You can also nest mixed list types\n## Like this\n\n! Ordered Lists\n# Ordered lists are pretty neat too\n# If you're handy with HTML and CSS you could customize the [[numbering scheme|http://www.w3schools.com/css/pr_list-style-type.asp]]\n## To nest, just add more octothorpes (pound signs)...\n### Like this\n* You can also\n** Mix list types\n*** like this\n# Pretty neat don't you think?\n\n! Tiddler links\nTo create a Tiddler link, just use mixed-case WikiWord, or use [[brackets]] for NonWikiWordLinks. This is how the GTD style [[@Action]] lists are created. \n\nNote that existing Tiddlers are in bold and empty Tiddlers are in italics. See CreatingTiddlers for details.\n\n! External Links\nYou can link to [[external sites|http://google.com]] with brackets. You can also LinkToFolders on your machine or network shares.\n\n! Images\nEdit this tiddler to see how it's done.\n[img[http://img110.echo.cx/img110/139/gorilla8nw.jpg]]\n\n!Tables\n|!th1111111111|!th2222222222|\n|>| colspan |\n| rowspan |left|\n|~| right|\n|colored| center |\n|caption|c\n\nFor a complex table example, see PeriodicTable.\n\n! Horizontal Rules\nYou can divide a tiddler into\n----\nsections by typing four dashes on a line by themselves.\n\n! Blockquotes\n<<<\nThis is how you do an extended, wrapped blockquote so you don't have to put angle quotes on every line.\n<<<\n>level 1\n>level 1\n>>level 2\n>>level 2\n>>>level 3\n>>>level 3\n>>level 2\n>level 1\n\n! Other Formatting\n''Bold''\n==Strike==\n__Underline__\n//Italic//\nSuperscript: 2^^3^^=8\nSubscript: a~~ij~~ = -a~~ji~~\n@@highlight@@\n@@color(green):green colored@@\n@@bgcolor(#ff0000):color(#ffffff):red colored@@\n
nslcutil.py
National Student Clearinghouse Enrollment Transmission checking program
Type the text for 'New Tiddler'
Type the text for 'New Tiddler'
[[overview]]
This nslcutil.py program is in a [[git repository|https://github.com/flintiii/nslcutil]]\n\nThis is based upon [[the Enrollment Report Programming and TestingGuide |https://www.studentclearinghouse.org/colleges/files/EnrollRept_ProgrammingandTestingGuide.pdf]]\n\n\n[[Functional Requirements|NSLCRequirements]]\n\nDownload the latest copy of this reference documentation from github with the following Linux command:\n\n{{{\nwget https://raw.githubusercontent.com/flintiii/nslcutil/master/2016-02-11_nslctiddler.html\n}}}
As of Wednesday, October 28, 2015\n\nAll ssh signins from outside goddard net arrive as 192.168.1.1\n\n[[activate.goddard.edu|https://activate.goddard.edu/]]\n\nis how we log in to create a new account\n\nAs of Thursday, October 22, 2nslc015 09:51 AM\n{{{\n 09:50:04 up 204 days, 1:08, 2 users, load average: 0.22, 0.31, 0.62\n}}}\n\n[[LVM Guidance |https://www.rootusers.com/how-to-increase-the-size-of-a-linux-lvm-by-adding-a-new-disk/]]\n\n{{{\nStarting Nmap 6.40 ( http://nmap.org ) at 2015-10-16 13:55 EDT\nNmap scan report for sis.goddard.edu (207.136.210.8)\nHost is up (0.0037s latency).\nrDNS record for 207.136.210.8: host8.goddard3.cust.sover.net\nNot shown: 996 filtered ports\nPORT STATE SERVICE\n22/tcp open ssh\n80/tcp open http\n443/tcp open https\n636/tcp open ldapssl\n}}}\n\n\n\n{{{\nroot@sis:~# df -h\nFilesystem Size Used Avail Use% Mounted on\n/dev/mapper/sis-root 11G 6.6G 3.8G 64% /\nudev 363M 4.0K 363M 1% /dev\ntmpfs 75M 268K 75M 1% /run\nnone 5.0M 0 5.0M 0% /run/lock\nnone 372M 0 372M 0% /run/shm\n/dev/sda1 228M 25M 192M 12% /boot\n//fileserver.campus.goddard.edu/id_photos 700G 461G 240G 66% /photos\n//fileserver.campus.goddard.edu/SIS_scans 700G 461G 240G 66% /scans\n//fileserver.campus.goddard.edu/SIS_uploads 700G 461G 240G 66% /uploads\n//fileserver.campus.goddard.edu/id_photos 700G 461G 240G 66% /photo_dir\n\n}}}\n\n\n\n{{{\nroot@sis:~# fdisk -l\n\nDisk /dev/sda: 23.6 GB, 23622320128 bytes\n255 heads, 63 sectors/track, 2871 cylinders, total 46137344 sectors\nUnits = sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 512 bytes\nI/O size (minimum/optimal): 512 bytes / 512 bytes\nDisk identifier: 0x00057b03\n\n Device Boot Start End Blocks Id System\n/dev/sda1 * 2048 499711 248832 83 Linux\n/dev/sda2 501758 25163775 12331009 5 Extended\n/dev/sda5 501760 25163775 12331008 8e Linux LVM\n}}}\n\nLVM configuration production\n{{{nslc\nroot@sis:~# vgdisplay\n --- Volume group ---\n VG Name sis\n System ID \n Format lvm2\n Metadata Areas 1\n Metadata Sequence No 3\n VG Access read/write\n VG Status resizable\n MAX LV 0\n Cur LV 2\n Open LV 2\n Max PV 0\n Cur PV 1\n Act PV 1\n VG Size 11.76 GiB\n PE Size 4.00 MiB\n Total PE 3010\n Alloc PE / Size 3003 / 11.73 GiB\n Free PE / Size 7 / 28.00 MiB\n VG UUID bNAhzb-JgoV-zvZP-vNGQ-nnSw-gEgc-PovhQk\n\n}}}\n\n{{{\nroot@sis:/var/www# apt-get -f install \nReading package lists... Done\nBuilding dependency tree \nReading state information... Done\nCorrecting dependencies... Done\nThe following packages were automatically installed and are no longer required:\n linux-headers-3.2.0-70-generic linux-headers-3.2.0-65-generic linux-headers-3.2.0-60 linux-headers-3.2.0-61\n linux-headers-3.2.0-63 linux-headers-3.2.0-64 linux-headers-3.2.0-70 linux-headers-3.2.0-65 linux-headers-3.2.0-72\n linux-headers-3.2.0-67 linux-headers-3.2.0-68 linux-headers-3.2.0-74 linux-headers-3.2.0-69 linux-headers-3.2.0-75\n linux-headers-3.2.0-76 linux-headers-3.2.0-77 linux-headers-3.2.0-68-generic linux-headers-3.2.0-60-generic\n linux-headers-3.2.0-76-generic linux-headers-3.2.0-63-generic linux-headers-3.2.0-74-generic linux-headers-3.2.0-69-generic\n linux-headers-3.2.0-61-generic linux-headers-3.2.0-77-generic linux-headers-3.2.0-64-generic linux-headers-3.2.0-72-generic\n linux-headers-3.2.0-67-generic linux-headers-3.2.0-75-generic\nUse 'apt-get autoremove' to remove them.\nThe following extra packages will be installed:\n linux-headers-3.2.0-92 linux-headers-3.2.0-92-generic linux-headers-generic\nThe following NEW packages will be installed:\n linux-headers-3.2.0-92 linux-headers-3.2.0-92-generic\nThe following packages will be upgraded:\n linux-headers-generic\n1 upgraded, 2 newly installed, 0 to remove and 164 not upgraded.\n7 not fully installed or removed.\nNeed to get 0 B/12.7 MB of archives.\nAfter this operation, 67.8 MB of additional disk space will be used.\nDo you want to continue [Y/n]? \n(Reading database ... 689118 files and directories currently installed.)\nUnpacking linux-headers-3.2.0-92 (from .../linux-headers-3.2.0-92_3.2.0-92.130_all.deb) ...\ndpkg: error processing /var/cache/apt/archives/linux-headers-3.2.0-92_3.2.0-92.130_all.deb (--unpack):\n error creating directory `./usr/src/linux-headers-3.2.0-92/drivers/infiniband': No space left on device\nNo apport report written because MaxReports is reached already\n dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)\nUnpacking linux-headers-3.2.0-92-generic (from .../linux-headers-3.2.0-92-generic_3.2.0-92.130_amd64.deb) ...\ndpkg: error processing /var/cache/apt/archives/linux-headers-3.2.0-92-generic_3.2.0-92.130_amd64.deb (--unpack):\n unable to create `/usr/src/linux-headers-3.2.0-92-generic/include/config/atm/solos.h.dpkg-new' (while processing `./usr/src/linux-headers-3.2.0-92-generic/include/config/atm/solos.h'): No space left on device\nNo apport report written because MaxReports is reached already\n dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)\nErrors were encountered while processing:\n /var/cache/apt/archives/linux-headers-3.2.0-92_3.2.0-92.130_all.deb\n /var/cache/apt/archives/linux-headers-3.2.0-92-generic_3.2.0-92.130_amd64.deb\nE: Sub-process /usr/bin/dpkg returned an error code (1)\n\n}}}\n\n{{{\nroot@sis:~# df -i\nFilesystem Inodes IUsed IFree IUse% Mounted on\n/dev/mapper/sis-root 720896 719481 1415 100% /\nudev 92908 436 92472 1% /dev\ntmpfs 95180 327 94853 1% /run\nnone 95180 5 95175 1% /run/lock\nnone 95180 1 95179 1% /run/shm\n/dev/sda1 124496 230 124266 1% /boot\n//fileserver.campus.goddard.edu/id_photos 0 0 0 - /photos\n//fileserver.campus.goddard.edu/SIS_scans 0 0 0 - /scans\n//fileserver.campus.goddard.edu/SIS_uploads 0 0 0 - /uploads\n//fileserver.campus.goddard.edu/id_photos 0 0 0 - /photo_dir\n\n}}}\n\nTry this program:\n{{{\n#!/bin/bash\n# count_em - count files in all subdirectories under current directory.\n# Thu Oct 22 06:45:16 EDT 2015 pflint\n# SOURCE: http://stackoverflow.com/questions/653096/howto-free-inode-usage\necho 'echo $(ls -a "$1" | wc -l) $1' >/tmp/count_em_$$\nchmod 700 /tmp/count_em_$$\nfind . -mount -type d -print0 | xargs -0 -n1 /tmp/count_em_$$ | sort -n\nrm -f /tmp/count_em_$$\n}}}\n\nbetter, try:\n{{{\nfor i in ./*; do echo -n $i" "; find $i -type f | wc -l; done |rev | sort -n | rev\n}}}\n\nTip:\n[[Read about php garbage collection|http://pim.famnit.upr.si/blog/index.php?/archives/172-Running-out-of-inodes,-no-space-left-on-device,-php-not-cleaning-sessions.html]]\n\nand see PHPgarbageSessions\n\n[[Changing SSL keys|ChangingSSLkeys]]\n\n[[Testing SSL keys|TestingSSLkeys]]\n\nInteresting warning \n{{{\nroot@sis:/etc/apache2# apache2ctl configtest \n[Wed Nov 18 11:24:07 2015] [warn] NameVirtualHost *:80 has no VirtualHosts\nSyntax OK\n}}}\n\nThis is where to try to find answer...\n|!Reference HTML|! Description|\n|[[basic goggle search |https://www.google.com/search?q=%5Bwarn%5D+NameVirtualHost+*%3A80+has+no+VirtualHosts&ie=UTF-8&sa=Search&channel=fe&client=browser-ubuntu&hl=en&gws_rd=ssl]]||\n|[[webmasterworld |https://www.webmasterworld.com/apache/3554465.htm]]||\n|[[Fix from serverfault |http://serverfault.com/questions/1405/apache-2-startup-warning-namevirtualhost-80-has-no-virtualhosts]]||\n|[[Ubuntu Forums|http://ubuntuforums.org/showthread.php?t=1480907]]||\n|[[Running Multiple Apps in Apache |http://mightydreams.blogspot.com/2011/03/running-multiple-applications-in-lan.html]]||\n|[[Hosting Multiple Websites |http://www.debian-administration.org/article/412/Hosting_multiple_websites_with_Apache2]]||\n\n
What NSLC needs is a text only spreadsheet.\nThe best way to do this is at import, select all the columns and then select from the drop down "Text"\n\nNSLCcurrentDataFlow\n\n[[Share for data|https://drive.google.com/a/goddard.edu/folderview?id=0B2TQ0PkLnoJYTVFvWEtBMS03UW8&usp=sharing_eid&ts=5661b057]]\n\n\nPossibilities include:\n* [[Ethercalc|https://ethercalc.net/]]\n\n[[Redmine Tickets|http://redmine.goddard.edu/search?q=nslc]]\n\nBe aware that in the [[instruction sheet|https://docs.google.com/spreadsheets/d/1vVU-1epH3evBxD-m6cbpse6sKDidi1KzSOMh2Npr5Ig/edit#gid=608968082&vpid=A2]] there is a [[google script|https://script.google.com/a/goddard.edu/macros/d/M1tKmNxvcgo70lrzmwqTDlPypkRWoM_M8/edit?uiv=2&mid=ACjPJvGZaNr9ZZ1OqBgxsES1TSKyZHGXebDpWmBs22QJaVsCk-NPMWNxq3vZP0XPW4I491jYuuzux2-m6tA2oIMHJBqB7sZhQGUZNHqiQyAhD8MoSVKD2FsOgJIZtu79N08geF428gDxzgU]] that does the final post processing
The current data flow is as folllows\n\n* Have privileges to the [["NSLC Reporting Google Drive"|https://drive.google.com/a/goddard.edu/folderview?id=0B2TQ0PkLnoJYTVFvWEtBMS03UW8&usp=sharing_eid&ts=5661b057]]\n* Open and follow [["Instructions"|https://docs.google.com/spreadsheets/d/1vVU-1epH3evBxD-m6cbpse6sKDidi1KzSOMh2Npr5Ig/edit#gid=608968082&vpid=A2]]\n* Logs into [[sis.goddard.edu|http://sis.goddard.edu]] with Registrar role. \n* Registrar runs report, [[nslc.php|nslc.php]]\n* Downloads, Open report in MS Excel, and edits spreadsheet.\n* Copies whole report and pastes it into [["Original From SIS"|https://docs.google.com/spreadsheets/d/1vVU-1epH3evBxD-m6cbpse6sKDidi1KzSOMh2Npr5Ig/edit#gid=897766620&vpid=A1]] a tab from the [["Instructions"|https://docs.google.com/spreadsheets/d/1vVU-1epH3evBxD-m6cbpse6sKDidi1KzSOMh2Npr5Ig/edit#gid=608968082&vpid=A2]]\n* Hits "Generate Report", which activates [[google script|]] that creates the report.\n
* Read line from csv\n* Create a line of spaces \n** string_val = "x" * 10 # gives you "xxxxxxxxxx"\n*** In [3]: outstr = 1250 * " "\n*** In [4]: len(outstr)\n\n* change this into a list\n** outlist = list(outstr)\n* change data into a list element\n** inlist = list("hello world")\n* place data in the list in the correct place\n** outlist[10:19] = inlist\n** "''.join(outlist)\n\n* take first vairable ljust padding with space it and place it in the string where it is supposed to go\n** take the next variable and place it in the string where it is supposed to go\n* loop till you run out of variables in the line\n* join the list and \n* write it out.
Type the text for 'nslc.php'
Google Script is basically Javascript.\n\nThe IDE for this is explained in this [[video|https://www.youtube.com/watch?v=ezbH4IJynhU]]\n\n{{{\n// Page vars\nvar max_rows = 2000;\nvar max_cols = 135;\nvar header_cols = 8;\nvar footer_cols = 11;\n\nvar ui = SpreadsheetApp.getUi();\n\n/**\n* Clears "Original from SIS" sheet\n*\n* @name clearSheet\n*/\nfunction clearSheet() {\n \n var ss = SpreadsheetApp.getActiveSpreadsheet();\n var thissheet = ss.getSheetByName("Original from SIS");\n thissheet.clear();\n}\n\n/**\n* Generates Tab Separated Values NSLC Report.\n* File will be created in user's "My Drive"\n*\n* @name generateReport\n*/\nfunction generateReport() {\n \n \n var response = ui.alert("Generate NSLC Report?", "This should only take a minute", ui.ButtonSet.YES_NO);\n \n // if yes\n if (response == ui.Button.YES) {\n \n // Get the active spreadsheet.\n var ss = SpreadsheetApp.getActiveSpreadsheet();\n \n // get Row 1 data from Header and Footer\n var header = getHeaderFooter(ss.getSheetByName("Header"),header_cols);\n var footer = getHeaderFooter(ss.getSheetByName("Footer"), footer_cols);\n \n // get data from Reformatted sheet\n var data = getData(ss.getSheetByName("Reformatted"));\n \n // test for real data returned\n if (data) {\n var doc = header + data + footer;\n \n // create a text file with the data\n var report_name = "nslc_003686_" + getToday() + ".txt"; \n var report = DriveApp.createFile(report_name, doc);\n \n // move file into NSLC Report/Final folder\n var report_folder = DriveApp.getFolderById("0B2TQ0PkLnoJYbFRVaC05NWRFZ1E");\n report_folder.addFile(report);\n \n // display "all done" message\n ui.alert('The NSLC Report is Complete', 'You can find the report here: https://drive.google.com/drive/?tab=mo#folders/0B2TQ0PkLnoJYbFRVaC05NWRFZ1E', ui.ButtonSet.OK);\n }\n }\n}\n\n/**\n* Gets header or footer content\n*\n* @name getHeaderFooter\n* @param {sheet} this_sheet. The sheet from which to pull the data.\n* @param {number} num_cols. The number of columns from which to pull data.\n* @return The data in TSV format.\n*/\nfunction getHeaderFooter(this_sheet, num_cols) {\n var range = this_sheet.getRange(1, 1, 1, num_cols);\n \n return format_tsv(range);\n}\n\n\n/**\n* Gets reformatted content\n*\n* @name getData\n* @param {sheet} this_sheet. The sheet from which to pull the data.\n* @return The data in TSV format.\n*/\nfunction getData(this_sheet) {\n \n // get column A\n var test_range = this_sheet.getRange(1, 1, max_rows, 1);\n var test_data = test_range.getValues();\n var test_data_len = test_data.length;\n var last_row = 0;\n \n // test for values to determine last row (can't use functions like getDataRange() because they will return in-cell formulas in addition to acutual values\n for ( var i=0; i<test_data_len; ++i ){\n if(isCellEmpty(test_data[i][0])) break;\n last_row = i + 1;\n }\n \n if (!last_row) {\n ui.alert('No Data was found on the Original from SIS sheet', 'Aborting.....', ui.ButtonSet.OK);\n return 0;\n } else {\n // use that range // get values\n var range = this_sheet.getRange(1, 1, last_row, max_cols);\n \n // format as TSV and return\n return format_tsv(range);\n }\n}\n\n\n/**\n* Reformat data into Tab Separated Values\n*\n* @name format_tsv\n* @param {range} range. A range object from the sheet from which to pull the data.\n* @return The data in TSV format.\n*/ \nfunction format_tsv(range) {\n \n var data = range.getValues();\n var formatted_data = "";\n for (var i = 0; i < data.length; i++) {\n var row = "";\n for (var j = 0; j < data[i].length; j++) {\n row += data[i][j] + "\sst";\n }\n formatted_data += row + "\ssn"; \n }\n \n return formatted_data; \n \n}\n\n\n/**\n* Tests whether parameter is empty\n*\n* @name isCellEmpty\n* @param {cell_data} string. The data to test.\n* @return Returns true if the cell where cell_data was read from is empty.\n*/ \nfunction isCellEmpty(cell_data) {\n return typeof(cell_data) == "string" && cell_data == "";\n}\n\n\n/**\n* Gets today's date\n*\n* @name getToday\n* @return Returns today's date in yyyymmdd form.\n*/ \nfunction getToday() {\n var today = new Date();\n var dd = today.getDate();\n var mm = today.getMonth()+1; //January is 0!\n var yyyy = today.getFullYear();\n \n if(dd<10) {\n dd='0'+dd\n } \n \n if(mm<10) {\n mm='0'+mm\n } \n \n return today = yyyy+mm+dd;\n}\n\n\n/**\n* Add menu item to run script\n*\n* @name onOpen\n* @param {event} e. \n*/ \nfunction onOpen(e) {\n \n // Get the active spreadsheet.\n var ss = SpreadsheetApp.getActiveSpreadsheet();\n var menuEntries = [ {name: "Clear 'Original from SIS' Sheet", functionName: "clearSheet"}, {name: "Generate Report", functionName: "generateReport"} ];\n ss.addMenu("NSLC", menuEntries);\n}\n}}}\n\nWhat can we learn about [[DriveApp.createFile]]
Data Distribution in export file:\n{{{\ncat good.txt |while read line; do echo "$line" |wc -c; done |sort -n |uniq -c\n}}}\n\nmaybe better with awk\n{{{\nawk '{ print length($0); }' "nslc_003686_202602_fixed (3).txt"\n}}}\n\n! Tab Approach\nThis will give you number of tabs line by line:\n{{{\nawk '{print gsub(/\sst/,"")}' infile > output.txt\n}}}\n\nThis will let you examine a particular tab value, in this case field 2 the SSN\n{{{\nawk 'BEGIN { FS = "\sst" } ; { print $2 }' \n}}}\n\nHeader 8 tabs\nData Lines 135 tabs\nFooter 11 tabs\n\nSearch cp d1_schema_complete_2015-12-09.csv schema.py ; sed 's/,//1' schema.py |sed 's/,/=[/1' | sed 's/,/","/g'| sed 's/$/"]/g' |sed 's/=\ss[/=\ss[\ss"/g'\n\nlinux library parsing fixed length mainframe\n\nhttp://www.linuxtopia.org/online_books/programming_books/python_programming/python_ch34s05.htmlcp \n{{{\nd1_schema_complete_2015-12-09.csv schema.py~ ; sed 's/,//1' schema.py~ |sed 's/,/=[/1' | sed 's/,/","/g'| sed 's/$/"]/g' |sed 's/=\ss[/=\ss[\ss"/g'# > schema.py\n}}}\n\n\nhttps://code.google.com/p/csvfix/\n\n* Save a fully quoted\n* Check for bad elements\n* Run this code\n{{{\ncp d1_schema_complete_2015-12-10-1.csv schema.py~ ;cat schema.py~ |sed 's/,//1' |sed 's/""//1' | sed 's/,/ = [/1' |sed 's/"//1' |sed 's/"//1' |sed 's/$/]/g' > schema.py \n}}}
\n* [[National Student Clearinghouse Enrollment Reporting Programming & Testing Guide|http://www.studentclearinghouse.org/colleges/files/EnrollRept_ProgrammingandTestingGuide.pdf]]\n* [[The part we are interested in|http://www.studentclearinghouse.org/colleges/files/EnrollRept_ProgrammingandTestingGuide.pdf#page=15]]\n* [[Header|http://www.studentclearinghouse.org/colleges/files/EnrollRept_ProgrammingandTestingGuide.pdf#page=14]]\n* [[Footer|http://www.studentclearinghouse.org/colleges/files/EnrollRept_ProgrammingandTestingGuide.pdf#page=31]]
Here is the script\n\n{{{\n ls -alt --full-time --time-style=long-iso . | grep $(date +%F) |cut -c 50- | grep -v ^"\ss." |head -1 |xargs awk 'BEGIN { FS = "\sst" } ; { print $51" "$52" "$53 }' \n}}}
\n[[standard input|https://ftps.nslc.org/]] ([[65.202.239.144|NSLCtransferScript]])\n[[standard error|https://secure.studentclearinghouse.org/stdreport/faces/stdreport?_afrLoop=225425469787007&_afrWindowMode=0&_adf.ctrl-state=16ms1ydp7g_4]]\n\nNSLC upload product is the Ipswitch "Move it" server.\nHere is the [[list of file transfer clients they support|http://docs.ipswitch.com/moveit/dmz8.0/help/anonuser/en/index.htm#23491.htm]]\n\nThe word from Daniel Hamilton\n{{{\nAnother unique feature of out SFTP software. You can connect with a key and it will get rejected the first time out, but it will be stored an one of us can "authorize" for use after the fact. I've done that for you here so follow up login attempts with this key won't be a problem.\n}}}
Here is the file compare code:\n\n{{{\n\n}}}
! To Operate\n# Run report in SIS\n# Download Report\n# Import into Excell\n# Modify to hearts content\n# Save as xlsx file\n# submit to nslcutil.py\n# done.\n\n! To Validate\n# take csv, tab, or txt file\n# submit to nslcutil.py -f\n# read result.\n
Options\n\n* [[openpyxl|https://pypi.python.org/pypi/openpyxl]] note must be version 2.3.2 or above\n** Download openpyxl-2.3.2.tar.gz\n** Figure out how to unzip on the windows platform\n** Test from python\n{{{\n if str(openpyxl.__version__) <> "2.3.2":\n}}}\nTADA\nThe error occurs if you start python in the following directory:\n{{{\nflint@seltzer:~/clients/goddard/sis/nslc/bin$ python\nPython 2.7.6 (default, Jun 22 2015, 17:58:13) \n[GCC 4.8.2] on linux2\nType "help", "copyright", "credits" or "license" for more information.\n>>> import openpyxl\nTraceback (most recent call last):\n File "<stdin>", line 1, in <module>\n File "/usr/local/lib/python2.7/dist-packages/openpyxl-2.3.2-py2.7.egg/openpyxl/__init__.py", line 26, in <module>\n from openpyxl.xml import LXML\n File "/usr/local/lib/python2.7/dist-packages/openpyxl-2.3.2-py2.7.egg/openpyxl/xml/__init__.py", line 27, in <module>\n LXML = lxml_available() and lxml_env_set()\n File "/usr/local/lib/python2.7/dist-packages/openpyxl-2.3.2-py2.7.egg/openpyxl/xml/__init__.py", line 11, in lxml_available\n from lxml.etree import LXML_VERSION\n File "serializer.pxi", line 4, in init lxml.etree (src/lxml/lxml.etree.c:193884)\n File "/usr/lib/python2.7/gzip.py", line 36, in <module>\n class GzipFile(io.BufferedIOBase):\nAttributeError: 'module' object has no attribute 'BufferedIOBase'\n>>> \n\nif you open in this directory:\n}}}\n\n\n* [[pythonexcells|http://pythonexcels.com/basic-excel-driving-with-python/]] uses COM/DCOM link
'12 January 2016'\nNSLCtransferScript\n\nNmap\n{{{\nStarting Nmap 6.40 ( http://nmap.org ) at 2016-01-12 08:55 EST\nNmap scan report for ftps.nslc.org (65.202.239.144)\nHost is up (0.027s latency).\nNot shown: 996 filtered ports\nPORT STATE SERVICE\n21/tcp open ftp\n22/tcp open ssh\n443/tcp open https\n3011/tcp closed trusted-web\n\n\n}}}
'12 January 2016'\n\n1. Re-patching the 12-29-15 submission ASAP\n2. Testing the 12-29-15 submission against past submissions\n3. Submitting this report.\n\n\n\n4. Completing the [[new program|NewProgram]]\n5. Testing it with sample data\n6. Submitting the new report with live data.\n
nslcutil.py\n\n{{{\nsed -i 's/\ss,\ss"/_/1' \nsed -i 's/_/__/1' d1_schema_meth_2016-01-24.csv\n}}}
'27 January 2016'\n\nNSLC Functional Requirements\n\n* converts xlsx file & transmistts to NSLC\n* Test the following fire formats:\n** xlsx \n** csv\n** tab\n....file types for data consistency\n* Convert csv to xlsx \n* Convert xlsx and csv to tab\n\nNSLCsources
Type the text for '8 February 2016'
Here are the steps for '29 January 2016'\n# Get the xlsx file from Josh\n# Open and remove the I row "degree concentration"\n# Save result as ".tab"\n# set filter to seperator is a tab\n# Copy up to Greg's script page\n# Run greg's report to get the A1 and the T1 records.\n# Download result\n# Gedit result\n# Gedit tab file\n# Copy the first and last line from gregs file into the tab file\n# save rename tab file\n# run {{{bin/chkt.sh <tab file name>}}} result:\n** number of tabs per line\n** No blank lines\n** 1 8\n** 1 11\n** 688 108\n** Nice numbers in the program 1 length field.\n# Upload by runing {{{bin/sf.sh <tab file name>}}}\n# Check [[log file|https://ftps.nslc.org/]] record number\n# Mail Geeta let her know...\n# save report to share and drive. \n\n
Here are the steps for '11 February 2016'\nNote:\nThis must be set up properly on the machine doing the work.\n\n# Download the "NSLC*.csv" file from SIS.\n# Convert the "NSLC*.csv" to "NSLC*.xlsx" using {{{bin/nslcutil.py -f NSLC*.csv NSLC*.xlsx}}}\n# Open with Microsoft Excell and remove the I row "degree concentration"\n** Save result as csv with name "NSLC*.tab"\n** set filter to separator is a tab\n# Check file using {{{bin/nslcutil.py -t NSLC*.tab }}}\n# Run {{{bin/nslcutil.py -a NSLC*.tab 2NSLC.tab --tab }}} to insert A1 header and T1 footer.\n# Run {{{bin/nslcutil.py -m 2NSLC.tab --tab }}} to send file to NSLC\n# Check [[log file|https://ftps.nslc.org/]] record number\n# Mail Geeta let her know...\n# save {{{2NSLC.tab}}} to share and drive. \n\nFinally be aware that with some further programming work this process can be further simplified and automated.\n