Knowledge Base

 
Reset Search
 

 

Article

NCOS: Troubleshooting Tools

« Go Back

Information

 
Content

NCOS: Troubleshooting Tools

Products Supported: AER31x0, AER21x0, MBR1400, MBR1200B, CBA850, CBA750B, IBR11x0, IBR6x0, IBR350. Click here to identify your router.

NCOS Version: 6.0 or later - for information on upgrading NCOS, click here.


Quick Links

Summary

GUI Tools

CLI Tools

NCM Tools

Status

Specific Logging

Informational

Testing

Related Articles


Summary

This article provides a summary of useful tools and commands found within the GUI, CLI and NCM for the purpose of determining the cause and finding resolution of issues experienced with Series 3 routers on NCOS 6 or later.


GUI Tools

Configuration Difficulty: Intermediate

Much of the information necessary for troubleshooting can be obtained directly from the NCOS GUI (graphical user interface), commonly referred to as "administrative pages".

To access troubleshooting tools available in the router's GUI, follow the steps listed below:

  • Step 1: Log into the router's Setup Page. For help with logging in please click here.
  • Step 2: Continue to follow the instructions listed below for accessing specific information.

Standard Router Log

  • Step 1: Click on the STATUS tab and select System Logs.
  • Step 2: Choose a method to review router activity:
    • Click Save Log to generate a text file with logging information. This option is useful when someone else needs to review the logged data, or current router events will need to be compared with another log at a later date.
    • Place a check next to Auto Update to tell the router to update the log with new events as they occur. This option is best used when reviewing router activity in real time.
    • Click Update to refresh the log to display new events since the page was loaded or last refreshed. This option is useful when events occur quicker than log entries can be read.
  • Step 3: Review our Router Log feature article for additional tips and tricks for using the standard router log.

User-added image

Support Log

The router Support Log bundles the system logs and some information about the current router configuration in a single file. This type of log is extremely helpful in troubleshooting issues after the fact because it provides a snapshot of what the router was doing at the time the problems occurred.

To generate the router Support Log, follow the steps listed below:

  • Step 1: Click on the SYSTEM tab, select Administration, and select System Logging.
  • Step 2: Click Save to disk next to Create support log to generate a new text file.
  • Step 3: Save the file to your computer, then send it as an email attachment to a Cradlepoint support representative for review.

User-added image

Debug Logging Level

At the DEBUG logging level, there is the possibility of over 1000 different new messages showing up in the router's log, depending on the services being used by the Cradlepoint router. It often helps to enable this option for a short time to get additional data during advanced troubleshooting.

Note: Using this logging level will impact router performance and over time can cause unexpected reboots or loss of functionality. This option should only be enabled at the request of an authorized Cradlepoint representative, and is recommended to be immediately be disabled after necessary logging information has been collected.

  • Step 1: Click on the SYSTEM tab, select Administration, and select System Logging.
  • Step 2: Click the drop-down arrow next to Logging Level and select DEBUG.

Note: Existing system logs will be unaffected by this change. Only events that occur after the change was made will trigger debug level messages.

User-added image

Verbose Modem Logging

Enabling any level of Verbose Modem Logging will write additional messages to the router's system logs regarding modem activity. This option is useful when there are problems with WAN connectivity. The types and amounts of messages you will see will depend on the modem you have connected and the modem's compatibility with the function. It often helps to enable this option for a short time to get additional data during advanced troubleshooting.

Note: Using this logging level will impact router performance and over time can cause unexpected reboots or loss of functionality. This option should only be enabled at the request of an authorized Cradlepoint representative, and is recommended to be immediately be disabled after necessary logging information has been collected.

  • Step 1: Click on the SYSTEM tab, select Administration, and select System Logging.
  • Step 2: Click the Level dropdown button next to Verbose modem logging.
  • Step 3: Select one or more options from the list of Error, Warn, or Info, as instructed by a Cradlepoint support representative.

Note: Existing system logs will be unaffected by this change. Only events that occur after the change was made will trigger debug level messages.

User-added image


CLI Tools

Configuration Difficulty: Expert

The command line interface of Cradlepoint routers offers commands that allow us to drill down to troubleshoot specific functions within the router.

To use the CLI, you will need to connect a client device to the Cradlepoint via SSH. Please refer to this article for specific instructions for enabling and using SSH with CradlePoint routers. Note: you can also use the CLI from the admin pages by going to System > System Control > Device Options and clicking the Device Console button

User-added image

The CLI of the Cradlepoint is separated in to “trees”. One of these trees it the “Status” tree. The Status tree allows you to see the the status of functions within the Cradlepoint similar to what you are able to see in the GUI. However, there are a few commands that you can use in the CLI to see statuses that are not available in the GUI.


NCM Tools

Configuration Difficulty: Intermediate
NetCloud Manager includes a number of basic but very important tools for diagnosing network problems. Often the first step in the troubleshooting process is to find out if the router is still online.

Devices

Routers:

  • Configuration
  1. Resume Updates: It is possible for a device's configuration changes to be "suspended." This means that a configuration change was sent to the device through NetCloud Manager that disabled the connection to NCM. Whenever this happens, the device rolls back to the previous configuration settings and it can no longer receive configuration updates. Select Resume Updates to restart configuration updates for the device. Note: You should first determine why the suspension occurred and correct the configuration.
  2. Clear: the NCM device configuration (including any local configuration); NCM group configuration remains.
  • NetCloud OS
  1. Retry Upgrade: When a device has a NetCloud OS error, select that device and click on NetCloud OS and then "Retry Upgrade" to attempt another NetCloud OS upgrade. (An error may have taken place because of a modem disconnect, for example).
  2. Cancel Upgrade: Stop an upgrade that is in progress and roll back.
  • Commands
  1. Reboot: Select one or more devices and then select Commands in the top toolbar and Reboot in the dropdown menu. Confirm that you want to reboot the selected device(s).
  2. Restore to Defaults: Select one or more devices and then select Commands in the top toolbar and Restore to Defaults in the dropdown menu to return the selected device(s) to factory defaults. You will be asked to confirm this action. Group and device configuration settings assigned through Cradlepoint NetCloud Manager will be reapplied once this operation is complete.
  3. Unregister: Select one or more devices and then select Commandsin the top toolbar and Unregister in the dropdown menu to remove the selected device(s) from NCM. You will be asked to confirm this action.
  4. WiFi Site Survey: Select a device and then select Commands in the top toolbar and WiFi Site Survey in the dropdown menu to search for nearby wireless access points. Use this to refresh the Devices → Rogue AP list and find rogue access points.
  5. Ping: Select a device and then select Commands in the top toolbar and Ping in the dropdown menu to open a popup window for a simple ping test, which measures round-trip message times.
  6. Traceroute: Select a device and then select Commands in the top toolbar and Traceroute in the dropdown menu to use the traceroute utility, which outputs the times and IP addresses for each hop along a route
  7. Speed Test: Select a device and then select Commands in the top toolbar and Speed Test in the dropdown menu to open a popup window for a speed test, which measures connection speed and outputs metrics such as Latency, Jitter, Loss, Throughput, etc.
  • Remote Connect
  1. Connect to Device UI: Connect to device's NetCloud OS to manage individual device.
  2. Console: Access device's Command Line Interface (CLI).
  • Export
  1. Router Logs: exports the logs of the selected router. Note: Log Reporting has to be enabled in the group settings for this to work.
  2. Activity Logs: exports activity log information for the selected router.
  3. Diagnostic Data: exports logs, full config and status after downloading them directly from the router over the stream connection.

Network Interfaces:​

  • Commands
  1. Manage Modem Firmware​: Manage the Modem firmware. Allows for checking, upgrading and switching firmware as applicable to the modem model.
  2. Manage Modem Firmware > Switch: Some modems can switch firmware to a different carrier. This feature, known as Multi-Carrier Software Defined Radio is available for specific modems.
  3. Manage Modem Firmware > Check:  Check whether a new modem firmware image is available. This will not cause a network interruption.
  4. Manage Modem Firmware > Upgrade: Apply a new firmware image to the modem. This action will cause a network interruption and may be best performed during low traffic/usage times.
  5. PUK Unlock: This will become enabled if the selected modem's SIM has been locked and therefore requires a PUK Code to be unlocked. The PUK unlock will not work if the modem is the only connection to the internet. There needs to be an alternative connection, such as a WAN ethernet port or dual modems.
Rogue AP: 
  • View a list of wireless access points that your devices have seen. Mark these as Known or Unknown. Use this list to search for rogue access points that could threaten your networks.

Groups

Router Groups:

  • NetCloud OS
  1. To upgrade NetCloud OS, select a group and click on NetCloud OS in the top toolbar. Note: NetCloud OS upgrades cause a device reboot.
  • Commands
  1. Reboot: Select one or more groups and then select Commands in the top toolbar and Reboot in the dropdown menu. Confirm that you want to reboot all the devices in the group(s).
  2. Restore to Defaults: Select one or more groups and then select Commands in the top toolbar and Restore to Defaults in the dropdown menu to return all the devices in the group(s) to factory defaults. You will be asked to confirm this action. Group and device configuration settings assigned through Cradlepoint NetCloud Manager will be reapplied once this operation is complete.
  • Settings
  1. Enable Log Reporting: (Disabled by default. Click to enable.) When this is enabled, you can export the router logs for devices within this group. Select a device from this group in the Devices page and click on Export and then Router Logs to export a CSV file of the router logs.
  2. Enable Usage Reporting: (Enabled by default. Click to disable.) When you run a report (see Reports), usage information will be displayed in time increments based on this interval setting. Reports are sent at least as often as the listed time; data usage reports are sent more frequently if there is heavy data usage. Changes to this setting will not take effect until a device reconnects to NetCloud Manager. You may force a re-connect by rebooting the device(s) if desired. (Default: 1 hour. Range: 5 minutes to 24 hours.)

Alerts & Logs

Alert Log:

Activity Log: 

  • See NetCloud Manager: Activity Log which describes how to access the NetCloud Manager (NCM) Activity Log, how to export it, and examples of what the log may contain.Activity Log entries are stored for 30 days.

Status

Router Statistics (Data from the STATUS > Internet > Statistics page in the router's GUI):

Product Info: get /status/product_info

LAN statistics: get /status/lan/stats

WAN statistics: get /status/wan/stats

Load Average (Average CPU utilization for the last 1, 5, and 15 minutes): get /status/system/load_avg

Memory Status: get /status/system/memory

Hotspot Status: get /status/hotspot


Specific Logging

These commands when increase logging for certain features within the Cradlepoint. These additional log entries will appear in the System Log in both the CLI and the GUI.

Note: Enabling additional levels of logging will impact router performance and over time can cause unexpected reboots or loss of functionality and should only be enabled at the request of an authorized Cradlepoint representative.

RADIUS: set /config/wlan/radius_verbosity 10 (any number between 1-10, the default is 6)

WLAN: set /config/wlan/wlan_verbosity 10 (any number between 1-10, the default is 6)

VPN: set /config/vpn/debug true

Routing Protocols

Web Filter: set /config/webfilter/logaccess true


Informational

SupportQA: This is a script that will issue preset commands in order to provide data for troubleshooting. The script will issue the following commands, for a definition of each command please refer to the appropriate command definition in this document.
f=resources()\nprint(f)\nf=threads(v=True)\nprint(f)\nf=devices()\nprint(f)\nf=lan()\nprint(f)\ nf=netfilter()\nprint(f)\nf=route()\nprint(f)\nf=wan()\nprint(f)\nf=wireless(v=True)\nprint(f)\nf=workqueue()\ nprint(f)\nf=get()\nprint(f)\n

diff: Show differences between the current configuration and the default configuration.

log: Show and manipulate the log system.

log [show|clear|level|msg] [SUBOPTONS...]
show [FILTER] [FILTERN] [[-bhi] -s SEARCH]

USAGE:

route: The route command can be used to show the current routing tables as well as make changes to the user defined static routing table.
Note: Only the user defined static routing table may be modified directly.

route [-v]
route add 172.0.0.100 dev lan
route add 10.0.0.0 netmask 255.0.0.0 gw 172.0.0.100

USAGE:

arpdump: The arpdump command shows the current arp table.

switch: Show ethernet switch status.

switch [-v] [show|set|clearstats]

USAGE:

tcpdump: Is a common packet analyzer that runs under the command line. It allows the user to intercept and display TCP/IP and other packets being transmitted or received over a network to which the computer is attached.

tcpdump [-aAdDeflLnNOpqRStuvxX] [-c count] [-E algo:secret] [-i interface] [-M secret] [-s snaplen] [-T type] [-y datalinktype] [expression]

wireless: Show the current wireless configuration and status.

wireless [OPTIONS...]

USAGE:

workqueue: Print current workqueue activities and schedules.

workqueue [-v] [-t TASKID] [-a] [monitor]

USAGE:

atterm: The atterm command can be used to get direct access to a modem's AT command channel. If a modem interface is not given then the system will pick the highest priority modem. To discover a modem's interface to use, use the command 'devices' and the value found under the 'SubIface' column.

atterm [interface] [-c ATCMD] [-t SOCKET_TIMEOUT]

threads: Show currently active threads.

devices: Display network devices connected to the router.

devices [-v]

USAGE:

help: Get help with a specific command, e.g., help CMD.

netfilter: Show info and debug from netfilter2 rule(s) and trigger(s).

netfilter [active|all|upnp|triggers|states] [input|routed|output] [ip/ip6] [-v] [-s] [-r RULE_INDEX] [-t TRIGGER_INDEX] [-s TATE_INDEX]

uptime: Show system uptime and load average.

inspect: Inspect a directory.

resources: Report the system resource usage.

ecm: Manage connection to the Cradlepoint Enterprise Cloud Manager.

ecm [alert|status|register|stop|start|restart|triggers]

USAGE:

Example:

State: Connected
Connected: 5:38:31
Router ID: 46697
Data Usage Input: 1.03M
Data Usage Output: 1.24M
Data Usage Period: 15 days, 15:07:03
Data Usage Monthly Estimate: 4.36M
Config Sync Status: ready
Recent Activity:
5:38:31 ago: Remote config read triggered
5:38:31 ago: Remote stream_client heartbeat triggered
5:38:31 ago: Remote config read triggered
5:38:31 ago: Remote config read of differences from default triggered
5:38:31 ago: Remote status read triggered

Testing

ping: Send a ICMP echo request to a remote computer
ping [-w DEADLINE] [-c COUNT] [-i INTERVAL] [-s SIZE] [-I SOURCE] HOSTNAME

ping6: Send a ICMPv6 echo request to a remote computer
ping6 [-w DEADLINE] [-c COUNT] [-i INTERVAL] [-s SIZE] HOSTNAME

traceroute: Trace the route to HOST
traceroute [-FIldnrv] [-f 1ST_TTL] [-m MAXTTL] [-p PORT] [-q PROBES] [-s SRC_IP] [-t TOS] [-w WAIT_SEC] [-g GATEWAY] [-i IFACE] [-z PAUSE_MSEC] HOST [BYTES]


Related Articles/Links


Published Date: 9/14/2015

This article not have what you need?  Not find what you were looking for?  Think this article can be improved?  Please let us know at suggestions@cradlepoint.com.

  • Connection Pulse: (Default: 120 seconds. Range: 10 seconds to 900 seconds) Devices check in with NCM according to this pulse interval. Note: Only shorten this interval if a device is having a hard time staying connected to NCM. Increasing the number of pulses could have a significant impact on data usage.
    • See NetCloud Manager: Activity Log which describes how to access the NetCloud Manager (NCM) Activity Log, how to export it, and examples of what the log may contain.Activity Log entries are stored for 30 days.
    • usage (in bytes/second): get /status/stats/usage
    • rf (measured in dBm): get /status/stats/rf
    • failover: get /status/stats/failover
    • rip: set /config/rip/debug true
    • ripng- rip for ipv6: set /config/ripng/debug true
    • bgp: set /config/bgp/debug true
    • ospf: set /config/ospf/debug true
    • FILTER can be one or more space-separated names or levels, e.g., log show wanmgr kernel DEBUG INFO
    • -b: bold new entries
    • -h HIGHLIGHT: same usage as searching but does not filter results
    • -i: makes the search case insensitive
    • -s SEARCH: can be any string to search for in the log message contents, e.g., log show -s Firmware
    • -f [LINES_OF_HISTORY]: follow mode with optional argument for number of lines of history to show
    • clear: erase all logs from memory
    • level [NEW_LEVEL]: view current level or set new log level to: critical, error, warning, info, or debug
    • msg [-l LEVEL] MESSAGE: write a message to the logging system; LEVEL defaults to Info
    • route add IP [netmask NM] [gw IP] [dev UID]: add new user defined static route
    • route del IP [netmask NM] [gw IP] [dev UID]: remove user defined static route(s)
    • -v: verbose output
    • show [port PORT_NUM]: only show the port specified
    • set PORT_NUM
    • [--mode=MODE]
      • [--gid=GROUP_ID]
      • [--link_speed=LINK_SPD]
      • [--mac=LINK_ADDR]
      • [--mode=MODE]
    • clearstats [PORT_NUM]
    • -w: Print information about enabled Access Point profiles
    • -d: Print information about enabled WiFi as WAN / WiFi Bridge profiles
    • -c: Print information about currently associated WiFi clients
    • -v: Print additional driver specific debug when available
    • -t [CHANNEL]: Set wireless channel
    • -k [MAC ADDRESS]: Kick wireless client
    • -s [OPTIONAL DWELL TIME]: Perform a wireless survey
    • -a [OPTIONAL DWELL TIME]: Do a survey and autoselect a channel
    • -r [RADIO]: Radio to operate on (0: first, 1: second)
    • -v: VERBOSE
    • -t TASKID: Only show information for a single task
    • -a: Show information for active tasks (ie. currently - executing)
    • devices [-v] [DEVICE_UID...]: Show one or more specific devices.
    • -v: Verbose output
    • status: [DEFAULT] Show current status information.
    • register [--username=USER --password=PW]|[--token_id=TID --token_secret=TS]: (Re)register
      • --username: The ECM username that should be used to authenticate.
      • --password: The ECM password that should be used to authenticate.
      • --token_id: [EXPERT] Token ID for token authentication mode.
      • --token_secret: [EXPERT] Token secret for token authentication mode.
    • unregister: Unregister this router from ECM and unregister.
    • stop: Manually stop the ECM client.
    • start: Manually start the ECM client.
    • restart: Manually restart the ECM client.
    • -F: Set the don't fragment bit
    • -I: Use ICMP ECHO instead of UDP datagrams
    • -l: Display the TTL value of the returned packet
    • -d: Set SO_DEBUG options to socket
    • -n: Print numeric addresses
    • -r: Bypass routing tables, send directly to HOST
    • -v: Verbose
    • -m: Max time-to-live (max number of hops)
    • -p: Base UDP port number used in probes (default 33434)
    • -q: Number of probes per TTL (default 3)
    • -s: IP address to use as the source address
    • -t: Type-of-service in probe packets (default 0)
    • -w: Time in seconds to wait for a response (default 3)
    • -g: Loose source route gateway (8 max)

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255